{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Environment and RL Agent Controller for a Thermostat\n",
    "\n",
    "```\n",
    "Author: Matt Pettis\n",
    "Github: mpettis\n",
    "Twitter: @mtpettis\n",
    "Date: 2020-04-27\n",
    "```\n",
    "\n",
    "This is a toy example of a room with a heater.  When the heater is off, the temperature will decay to 0.0, and when it is on, it will rise to 1.0.  The decay and rise is not instantaneous, but has exponential decay behavior in time given by the following formula:\n",
    "\n",
    "    temperature[i + 1] = heater[i] + (temperature[i] - heater[i]) * exp(-1/tau)\n",
    "\n",
    "Where:\n",
    "\n",
    "    temperature[i] is the temperature at timestep i (between 0 and 1).\n",
    "    heater[i] is the applied heater, 0 when not applied, 1 when applied.\n",
    "    tau is the characteristic heat decay constant.\n",
    "\n",
    "So, when the heater is off, the temperature will decay towards 0, and when the heater is on, it will rise towards 1.  When the heater is toggled on/off, it will drift towards 1/0.\n",
    "\n",
    "Here is a sample plot of what the temperature response looks like when the heater is on for a while, then off for a while.  You will see the characteristic rise and decay of the temperature to the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x262dcb58bc8>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXyU1dn/8c+VZRKSCUsyQRFkUcMSyiIEBCyCYpFNUFwKalWq8KCiba2t/J4+tVrrY1tbtSLKg1VRq1JRqYgoChWlKquCshMWIaBAwhoCZDu/P84kjmGSTJKZuWcm1/v1mhdJ7jszV27Hb07OfRYxxqCUUir6xTldgFJKqeDQQFdKqRihga6UUjFCA10ppWKEBrpSSsWIBKde2OPxmPbt2zv18kopFZVWr16db4zJ9HfMsUBv3749q1atcurllVIqKonI19Ud0y4XpZSKERroSikVIzTQlVIqRjjWh66Uim0lJSXk5eVx8uRJp0uJSsnJybRp04bExMSAv0cDXSkVEnl5eaSlpdG+fXtExOlyoooxhoKCAvLy8ujQoUPA31drl4uIPCci+0VkXTXHRUSeEJFcEflSRHrVoW6lVIw6efIkGRkZGub1ICJkZGTU+a+bQPrQZwHDajg+HMjyPiYBT9epAqVUzNIwr7/6XLtau1yMMR+LSPsaThkDvGjsOrzLRKS5iLQyxnxT0/PuO3qSR9/fXKdilWooEeGqXm1om5HidClKBV0w+tBbA7t9Ps/zfu20QBeRSdhWPK4zz2Pah7lBeHmlAmcMHDtZyn2XZztdioogS5YsweVyMWDAAABmzJhBSkoKN954o8OV1U0wAt3f3wV+d80wxswEZgLk5OSYVQ+PDMLLKxW4wY98yIHCU06XoSLMkiVLcLvdlYE+efJkhyuqn2CMQ88Dzvb5vA2wNwjPq1TQedxJ5B/TQG8srrjiCnr37k3Xrl2ZOXMmAO+99x69evWiR48eDBkyhJ07dzJjxgwee+wxevbsydKlS7n//vv5y1/+AsCaNWvo168f3bt358orr+TQoUMADB48mHvvvZe+ffvSsWNHli5d6tjPWSEYLfR5wBQRmQ1cAByprf9cKadkuF1sP3Dc6TIanQfeXs+GvUeD+pzZZzXld5d3rfGc5557jvT0dE6cOEGfPn0YM2YMEydO5OOPP6ZDhw4cPHiQ9PR0Jk+ejNvt5p577gFg8eLFlc9x4403Mm3aNAYNGsR9993HAw88wOOPPw5AaWkpK1asYMGCBTzwwAMsWrQoqD9jXdUa6CLyKjAY8IhIHvA7IBHAGDMDWACMAHKBImBCqIpVqqE87iRW7DjodBkqTJ544gnmzp0LwO7du5k5cyYXXXRR5dju9PT0Gr//yJEjHD58mEGDBgFw0003cc0111QeHzt2LAC9e/dm586dIfgJ6iaQUS7jazlugDuCVpFSIeRxJ3GoqITSsnIS4nXli3CprSUdCkuWLGHRokV89tlnpKSkMHjwYHr06MHmzcEbXZeUlARAfHw8paWlQXve+tJ3tIp9xsDRvbB9Cf0PzaMVBRw8Xux0VSrEjhw5QosWLUhJSWHTpk0sW7aMU6dO8dFHH7Fjxw4ADh60f62lpaVx7Nix056jWbNmtGjRorJ//KWXXqpsrUcinfqvYkdZCRzcAflbIH8z5G+FA95/i+3/rP2AqYkDOFA4hpZNk52tV4XUsGHDmDFjBt27d6dTp07069ePzMxMZs6cydixYykvL6dly5Z88MEHXH755Vx99dW89dZbTJs27XvP88ILLzB58mSKioo455xzeP755x36iWontsck/HJycoxucKHq5dQxb2hXBPYW+zi4Hcp9/uxNOws8WZDZCTwdwdORgqV/x7X9A74Yt4qLurRx7mdoBDZu3EiXLl2cLiOq+buGIrLaGJPj73xtoavIduxbyFsJu1fAt1/aED+657vjEg/p59jQ7jzKBndmR8jIguSmpz1dyaGjZOyYR/yOJdDlhvD9HEqFgQa6ihylxTa0KwI8bxUc2WWPxSXCGV2h/cDvt7pbdIAEV8Avkdr5Eo7OS8Gz6z1AA13FFg105Zwje2x4Vzz2roEy76Sfpm2gTQ70mwxt+sCZ3SGx4X3e7pQU3jK9uezAEvsLpA6/DJSKdBroKjxKTtrW9+4V3wV4RddJfBKcdT70nQhn97UB3vSskJQhInyW9EOuKFkKOz+G8y4Nyeso5QQNdBUaxsC+9bD5Xdj6PnyzBsq8QwWbt4W2/aBNXzi7D5zRLawt5W3NLuBEQROabJinga5iiga6Cp6yUtj1KWxaAJsXwOGvAYHWvaHfbTbA2/SBtDMcLbOZ283yI30YvGk+jHwU4vV/AxUb9J2sGubUMchdbAN8y0I4edh2oZx7MQy8GzoOdzzAq/K4k3i3vC+Diz62v4A6XOR0SUoFhQa6qruj38CWd21LfMdHtiulSTp0GgGdR8C5l4Ar1ekqq+VJc/HCia78MaUJsuEtDfRGwBiDMYa4uNieHB/bP50KDmNg3wb4+BF45hJ4tDPM/wUU5ELfSXDzArhnK1z5NHS5PKLDHGwLvbA8iZIOl8DG+VBe7nRJKgR27txJly5duP322+nVqxcvvfQS/fv3p1evXlxzzTUUFhYCMHXqVLKzs+nevXvlaos333wzkydPZuDAgXTs2JH58+cDdp/UCRMm0K1bN84//3w+/PBDAGbNmsXYsWMZNmwYWVlZ/PrXvwagrKyMm2++mR/84Ad069aNxx57DIBt27YxbNgwevfuzcCBA9m0aVNQfmZtoSv/jLHjwNfPtd0ph+zaF7TuDZf8FjqPhMzOEIV7RnrcdkGlgnbDabX1HchbYW/SqtB5dyp8+1Vwn/PMbjD8jzWesnnzZp5//nl+//vfM3bsWBYtWkRqaip/+tOfePTRR5kyZQpz585l06ZNiAiHDx+u/N6dO3fy0UcfsW3bNi6++GJyc3OZPn06AF999RWbNm1i6NChbNmyBbDrpn/xxRckJSXRqVMn7rzzTvbv38+ePXtYt24dQOXzT5o0iRkzZpCVlcXy5cu5/fbb+fe//93gS6KBrr6v9JQN8eUzYO8XEO+CDoPgwrtsf3jTVk5X2GAZbjuiZlf6D2kV74INb2mgx6h27drRr18/5s+fz4YNG7jwwgsBKC4upn///jRt2pTk5GRuvfVWRo4cyahRoyq/99prryUuLo6srCzOOeccNm3axH/+8x/uvPNOADp37ky7du0qA33IkCE0a9YMgOzsbL7++mu6du3K9u3bufPOOxk5ciRDhw6lsLCQTz/99HvL8J46FZxNVzTQlXXsW1j1nH0cP2BnYY78K3S71u8U+miW6W2hf1vsgnOHwIZ5cNn/RuVfG1GjlpZ0qKSm2u4/Yww/+tGPePXVV087Z8WKFSxevJjZs2fz5JNPVraUpcr7QUSoae2riqV04bvldFu0aMHatWtZuHAh06dP57XXXuPxxx+nefPmrFmzJhg/4vdoH3pjl7ca3pgIj/0APvqz7VL5yVy4YwX0uTXmwhy+63LJLyyG7NFwNA/2fO5wVSqU+vXrxyeffEJurt2YvqioiC1btlBYWMiRI0cYMWIEjz/++PdCds6cOZSXl7Nt2za2b99Op06duOiii3j55ZcB2LJlC7t27aJTp07Vvm5+fj7l5eVcddVVPPjgg3z++ec0bdqUDh06MGfOHMD+slm7dm1Qfk5toTdGpcW2m2H5DNizClxpNrz7ToSMc52uLuSaNUkkIU4oKDwFvYdDXAJsfAva9Ha6NBUimZmZzJo1i/Hjx1d2b/zhD38gLS2NMWPGcPLkSYwxlTctATp16sSgQYPYt28fM2bMIDk5mdtvv53JkyfTrVs3EhISmDVr1vda5lXt2bOHCRMmUO698f7www8D8PLLL3Pbbbfxhz/8gZKSEsaNG0ePHj0a/HPq8rmNSeF+WD0LVj4Lhd9CxnnQ97+g53hISnO6urC64H8XMahjJn++uge8NBYOboO71mi3SxBF8/K5N998M6NGjeLqq692tA5dPledbu8XsPz/YN0bdsz4eT+CC6bb8eIxPi63Oh53ku1yAcgeA2/fZUdhtOrubGFKNYAGeqwqK4GNb9sg370MXG7ofbMdN+7Jcro6x2W4k8gv9I4s6DwS5v/cdkNpoCvsuPJopIEea8rL4PMX4KNH4Nheu174sD9Cz+sguZnT1UUMj9tF7j7vHpKpHmj/Qxvol/yPdrsEkTHmtNEiKjD16Q7XQI8le1bDO7+0XSxt+8OoxyDrRxAX73RlESfT2+VSGThdRsOCe+DAJmgZnf2+kSY5OZmCggIyMjI01OvIGENBQQHJyXXbA0ADPRYUHYTFD8DqF8DdEsb+HbpdrS3NGnjcSRSXlXPsVClNkxPtkgULfmXHpGugB0WbNm3Iy8vjwIEDTpcSlZKTk2nTpm773mqgR7Pyctu9svgBOHkU+t8Bg+6NybHjweZJs7NF84+dsoGedqadLbrhLRh8r8PVxYbExEQ6dOjgdBmNSuMc4hAL9qyGvw+xN/NaZsPk/8BlD2mYB+h7k4sqZI+B/eshP9ehqpRqGA30aFN0EN7+OTwzxG7hNvYZuPkdOCPb6cqiSkZqRaD7rKHR5XL778a3HKhIqYbTLpdoUV4OX7wIix6Ak0eg3+0weKq2yOupssvFN9CbtYHWObYffeAvHapMqfrTQI8Gez63IzD2rIa2A2DkX+CMrk5XFdXSU1yIVOlyAbu2ywf3waGvoUU7Z4pTqp60yyWSFR20G0k8cwkc3g1XzoQJCzTMgyAhPo70FNf3W+hghy8CbJwX/qKUaiBtoUei8nL44iVYdL+3e+U2b/eKTgwKJo87ifxjVQI9vQOc2d12uwy405nClKonDfRIs/cLeOceuwpi2/4w4i9w5g+criomZbj9tNDBjnb594NwZA80ax3+wpSqp4C6XERkmIhsFpFcEZnq53gzEXlbRNaKyHoRmRD8UhuBlX+3o1cO7/J2r7yrYR5C31ugy1f2GPvvpvnhLUipBqo10EUkHpgODAeygfEiUnWM3B3ABmNMD2Aw8FcRcQW51thVVmpnKb7zSzjvUpiyEnr8WGd6hpjHd4Gu7x3IsmP7N+jwRRVdAmmh9wVyjTHbjTHFwGxgTJVzDJAmdsEGN3AQKA1qpbHqxGF45RpYMRP6T4Hxr0KT5k5X1Sh40lwUFZdRVOznrdplNHz9qV1DXqkoEUigtwZ2+3ye5/2aryeBLsBe4CvgZ8aY8qpPJCKTRGSViKzS9R2Ag9vh2R/Bjo9h9DQ701MX0gqbitmiBdV2uxi7BLFSUSKQQPf3d3/VdR0vA9YAZwE9gSdF5LQZL8aYmcaYHGNMTmZmZp2LjSk7P7H95ccPwE/+Bb1udLqiRqdis+gD/rpdWnaxOzrp8EUVRQIJ9DzgbJ/P22Bb4r4mAG8aKxfYAXQOTokx6POX4MUxdh3uWxdDh4FOV9QoZbi/W6DrNCK2lb5jqZ0PoFQUCCTQVwJZItLBe6NzHFC12bILGAIgImcAnYDtwSw0JpSXwfv/A/Om2BC/5YNGsSlzpPK7QJevLqPBlMGmd8JYlVL1V2ugG2NKgSnAQmAj8JoxZr2ITBaRyd7THgQGiMhXwGLgXmNMfqiKjkqnjsHs6+HTadBnIlw3R29+Oqyyhe6vywWgVQ9o3k5Hu6ioEdDEImPMAmBBla/N8Pl4LzA0uKXFkMO74JVxdjecEX+BvhOdrkgBSQnxNE1OoKC6QBexa7ssm2FHI+kvYBXhdC2XUNu9wq7FciQPrp+jYR5hPGnVTC6qkH0FlJfAlvfCV5RS9aSBHkpfvgazRoLLDbcugvOGOF2RqsLjTvI/yqXCWb2gaWu7totSEU4DPRTKy2Hxg/DmRGjTFyb+GzI7Ol2V8sNT3XouFeLi7M3R3EX2PohSEUwDPdiKj8Ocm2DpX+zY8p/MhZR0p6tS1fC74mJV2aOh7BRsfT88RSlVTxrowXR0Lzw/3M4uHPoQXP4EJOiSNpHM407i6MlSiktPm9j8nbMvAPcZ2u2iIp4GerDsXWNvfhZsg/GzYcAUXVwrClRO/z9eU7dLPHQeZVvoxUVhqkyputNAD4aD2+EfYyEuAW55HzoNc7oiFSBP5WzRGka6gJ01WlIE2xaHoSql6kcDvaFOHIKXrwVTDje+pdvDRZmMytmitfSjt7sQmqTrJCMV0TTQG6K0GP75Ezj8NYx7RafxR6EaF+jyFZ8AXUbB5vegtJZzlXKIBnp9GQPzfw47l8LoJ6HdAKcrUvXgSatl+r+vLmOg+Bhs+zDEVSlVPxro9bX0r7DmZRg01e4upKJSiiuBFFe8/zXRq+pwkd2oW5fUVRFKA70+1r1pNxHudg0MPm2LVRVlqt2KrqoEF3QaYVdfLCsJfWFK1ZEGel3tXglzJ8PZ/WxXiw5NjHq1zhb1lT0GTh62u0wpFWE00Ovi0E54dRw0PcveBE1MdroiFQQZ7qTahy1WOOdiuzaPjnZREUgDPVAnDtvhieWldtXE1AynK1JBEnCXC9hf4h2HebtddB90FVk00ANRVgKv3WgnEP34H+DJcroiFUSZbhcHi4opLath+r+vrldAUb4uqasijgZ6bYyBd+6GHR/B5X/T/T9jkCctCWPgUFGANzo7Dofmbe3uU0pFEA302nz6BHz+Igz8JZx/vdPVqBDwBDpbtEJ8AvS7A3YvsxuYKBUhNNBrsmEefPA76DoWLv4fp6tRIVLnQAc4/wZIbm5/4SsVITTQq7NnNbw5CdrkwBVP2Y0OVEyqdbNof5Lc0OcW2DjfrrCpVATQlPKnYlNnd0sY9yokNnG6IhVClS30QIcuVug7CeIT4bPpIahKqbrTQK/q5BF45cd2AabrXgN3ptMVqRBrmpyAKz6O/JrWRPcn7Uzo/mO7BMTx/NAUp1QdaKD7KiuFORMgfwtc+wK07Ox0RSoMRMTOFq1rCx1gwJ1QehJW/j34hSlVRxroFYyBd39lNzAY+Sice7HTFakw8qTVYXKRr8xOdqLRiplQciL4hSlVBxroFZY9Bauegwt/Br1vcroaFWYZqXVYz6WqAXdCUQGseSW4RSlVRxroYKdxL/wNdBkNQ+53uhrlgDpN/6+q3YVwVi97c7S8LLiFKVUHGuj7N8Ibt0LrXnDl/+nwxEbKk5ZEQWEx5eWm7t8sYlvpB7fB5gXBL06pADXu9Covh7d/BgnJdniiK8XpipRDPO4kSssNR0/Wc53zLqOheTtdDkA5qnEH+pqXYfdyGPogpJ3hdDXKQZ76TC7yFZ8A/e+w76ddy4NYmVKBa7yBXnQQPrjPblTR4zqnq1EOq9wsuj5DFyvocgDKYQEFuogME5HNIpIrIn73XBORwSKyRkTWi8hHwS0zBD64z04iGvWo9psrMuqznktVrlToc6u9yZ6fG6TKlApcrUkmIvHAdGA4kA2MF5HsKuc0B54CRhtjugLXhKDW4Nm1DL54CfrfDmd0dboaFQEa3OVS4YL/sssBLNPlAFT4BdI07QvkGmO2G2OKgdnAmCrnXAe8aYzZBWCM2R/cMoOorATm3w1N28Ag3eBZWS1SXMTHScMD3d0SeoyzY9ILDwSnOKUCFEigtwZ2+3ye5/2ar45ACxFZIiKrReRGf08kIpNEZJWIrDpwwKE3+/L/g/3rYfgf7Yp5SgFxcUJ6qouCwgb0oVfor8sBKGcEEuj+trWvOlg3AegNjAQuA34rIh1P+yZjZhpjcowxOZmZDix6dWQPLHkYsoZC51Hhf30V0Ro0uchXZke7q9GKmVBc1PDnUypAgQR6HnC2z+dtgL1+znnPGHPcGJMPfAz0CE6JQfTeVLvJ8/A/28kgSvnwuF0cCEYLHeDCu+DEQVirywGo8Akk0FcCWSLSQURcwDhgXpVz3gIGikiCiKQAFwAbg1tqA239ADbOg4vugfQOTlejIpDHnUT+sSC00AHa9ofWOfDpk7ocgAqbWgPdGFMKTAEWYkP6NWPMehGZLCKTvedsBN4DvgRWAH83xqwLXdl1VHICFtwDGVkw4C6nq1ERyuO2C3QZU4/p/1VVLAdwaIcdxqhUGCQEcpIxZgGwoMrXZlT5/BHgkeCVFkRLH4VDO+HGeZCQ5HQ1KkJ53EmcKi3neHEZ7qSA/teoWZfLoUV7O9Goy+XazadCLvZn1OTnwiePQ7dr4JxBTlejIth3W9EFqdslLh76T4G8lXZJAKVCLLYD3Rh4525IaAJDH3K6GhXhPGlBmC1aVc/roUk6fKLLAajQi+1AX/cG7PgIhvxWF99StcpIDdJsUV+uFLscwOYFkL81eM+rlB+xG+gnj8DC/4ZWPSHnp05Xo6JApreFHrShixX6ToJ4F3z2ZHCfV6kqYjfQ//0QFO6HUY/ZvkylapFe0UIPVh96BXcm9BwPa17V5QBUSMVmoO9dAyufgT632J2IlApAYnwcLVISKTge5EAHe3O0rNjOHlUqRGIv0MvLYP4vIMUDl/zW6WpUlLGTi4Lc5QLgyYJOI2xDQ5cDUCESe4G++nnY+zlc9hA0ae50NSrKBG09F38uvAtOHLI7ZSkVArEV6IX7YdHvof1AO+5cqTrK8M4WDYmzL4A2fezNUV0OQIVAbAX6+7+FkiIY+ajOylP1YlvoIehyAe9yAHfZWcsb3w7Na6hGLXYCfcdS+HK2/bM287SVe5UKSGZaEoWnSjlZEqIWdOeRkH6OXQ4gGGvGKOUjNgK9tBje+SU0bwsD73G6GhXFgrYVXXXi4qH/HbBnNez6LDSvoRqt2Aj0z6ZB/mYY/oidmadUPVWu5xKqbheAHtdBSgZ8Oi10r6EapegP9EM74aNH7A5EnYY5XY2KckFfoMsfVwr0mWiXAziwJXSvoxqd6A/0d6fam03D/uh0JSoGZIS6y6VC34mQkAyf/i20r6MalegO9E3vwJZ3YfBUaH527ecrVYvvulxCHOipHrvG0BcvQ96q0L6WajSiN9DLSuweoS2zod/tTlejYkRyYjxpSQmh7UOvcPF/Q9OzYN5d9v2sVANFb6BvnAeHd9np/fGJTlejYognLYSzRX0lpcHIv8L+9fCJdr2ohoveQF/2NLToAB31RqgKLk8oZ4tW1Wk4ZI+Bj/4MBdvC85oqZkVnoO9eabf16ncbxEXnj6AiV0ZqCGeL+jP8z/YG6ds/08lGqkGiMw2XPQVJTaHndU5XomKQJy2MLXSAtDPhRw/AzqW6cJdqkOgL9CN5sOEt6HWj7YNUKsg87iQOF5VQUlYevhftdRO0HQALf2MXmVOqHqIv0FfMBIzd1kupEKgYunjweBi7XeLi4PK/2cXl3psavtdVMSW6Ar34OKyeZWeFtmjndDUqRlUE+oFQzhb1J7OjXYto3Ruw5f3wvraKCdEV6GtftZs/97/D6UpUDMtMC9NsUX9++AvI7Azv3A2nCsP/+iqqRU+gl5fboYpnnW83ClAqRDJSw7BAV3USXLbr5chu+PB/w//6KqpFT6DnLoKCXOh3h25eoULKkxam6f/VadsPcm6B5U/bZXaVClD0BPqypyCtlZ2EoVQIpbriSU6MC+2Ki7W59HfgPgPm/UyXBVABi45A37cBtn8IfW61f5IqFUIigsedREE4R7lUldwMRjwC+76Cz6Y7V4eKKtER6MuftjPpcn7qdCWqkbB7izrYQgfocrkd0bXkYTi43dlaVFSI/EA/ng9r/wk9xkFKutPVqEbC404K/7BFf0Y8AvEumP8LXRZA1SqgQBeRYSKyWURyRaTaWQ8i0kdEykTk6qBVuOp5KDsFF9wWtKdUqjZ2gS4Hu1wqND3L9qdvXwJrZztdjYpwtQa6iMQD04HhQDYwXkSyqznvT8DCoFVXWgwrn4Fzh0DLzkF7WqVq43EncfD4KcrKI6BV3Pundqjuwv+2f7EqVY1AWuh9gVxjzHZjTDEwG/A31ORO4A0geAtRrJ8Lhft0AwsVdh63i3IDh4sioJUeFweXPwGnjtlQV6oagQR6a2C3z+d53q9VEpHWwJXAjJqeSEQmicgqEVl14MCBml/VGFg2HTwd4dxLAihTqeD5bix6BAQ62L9QB94NX/7TzslQyo9AAt3fLJ6qf4c+DtxrjCmr6YmMMTONMTnGmJzMzMyaX3XXZ/DNWl3zXDkibHuL1sXAX0JGlr1BWnzc6WpUBAokKfMA3x2Y2wB7q5yTA8wWkZ3A1cBTInJFgypb9hQkN4fu4xr0NErVh8ft4Hou1UlIgtFP2K0XlzzsdDUqAgUS6CuBLBHpICIuYBwwz/cEY0wHY0x7Y0x74HXgdmPMv+pd1aGdsOkdyJkArpR6P41S9eXYiou1aTcAet9sJxvtXeN0NSrC1BroxphSYAp29MpG4DVjzHoRmSwik0NS1fKZIHHQZ2JInl6p2jRrkkhivEROH7qvSx+A1Ex4+y4oK3W6GhVBEgI5yRizAFhQ5Wt+b4AaY25uUEUnj8LnL0L2FdCsde3nKxUCIkJGahIFkdTlUqFJc7sP6Zyb7CzqAXc6XZGKEJF3t3HNy1B8TIcqKseFfW/RusgeA51G2CV2D+10uhoVISIr0MvLYPkMO4miTW+nq1GNnF3PJQK7XMAuIT3iEds1Of9uXRZAAZEW6Fves62NfjrNXzkvIzUCFuiqSbM2MOR3sG0xrHrW6WpUBIisQP/sKWh2NnS+3OlKlMKT5qKgsBgTya3fPrdA1lBY8CvY/J7T1SiHRU6gf7MWvv4P9J0E8QHdq1UqpDLdSRSXlXP0RASPJImLh6ufhzO7w+sTIE93OGrMIifQl82AxFTodaPTlSgF+MwWPR7B3S4ASW64fg64W8Ir10DBNqcrUg6JjEA/tg/WvQ7nX2+HZCkVASoDPdImF/njbgnXv2Fvjv7jKiisZa0kFZMiI9BXPQtlxXBBaOYpKVUfnrSK6f8ROtKlKs95cN1rcOxbeOVaXe+lEXI+0EtOwspnoeMwyDjX6WqUqpSRGoELdNXm7D5w9XPwzRqYM0FnkjYyzgf6V3OgKF8nEqmIk57qIk6iLNABOo+wY9S3LoR3dIx6Y+LscBJjYNnT0LIrdLjI0VKUqio+TkhPjZCt6Oqqz61wZA/851E7Xp9+rn4AAA/ZSURBVH3Qr52uSIWBs4G+42PYvx5GP2lnvikVYexs0ShroVcYch8c3QsfPmT3Jj3/BqcrUiHmbKAvewpSPNDtGkfLUKo6UR3oIjB6GhR+C/PuAveZkHWp01WpEHKuD730lJ3q3+cWSEx2rAylapLhjuAFugKR4IJrX4IzsuG1G2HvF05XpELIuUA/fgDiXZBzi2MlKFUbjzuJ/GNR2IfuK7kpXDcHUtLh5Wt1dcYY5lygFx2EH1wNaWc4VoJStfG4kzhRUsbxU1E+/K9pK7jhDTvf4x9XwfECpytSIeBcoJsyXVVRRbyKvUULonGkS1WZnWD8bDi8G14dByUnnK5IBZlzgZ7igVbdHXt5pQLhSfPuLRrN/ei+2vWHq56BvJXwxq12DwIVM5wL9OZnO/bSSgUq0x2Fs0Vrkz0Ghv0RNs2Hd+/ViUcxRNepVaoGGe6K9VxiKNAB+k2Go3nw6TS7d+8Pf+F0RSoINNCVqkHlei7RPtLFn0t/byceLbofmraG7tc6XZFqIA10pWrgSoijWZPE2GuhA8TFwRVPQ+F++NftdgnecwY7XZVqAOcX51IqwnncLgoifZOL+kpIgh//AzxZ8Op1sHG+0xWpBtBAV6oWMTG5qCZNmsMNb9phjf+8Hpb8EcrLna5K1YMGulK18KRF8XougWraCia8Cz3Gw5KH4bWfwKljTlel6kgDXalaeFJdsTMOvSaJybZP/bKHYfO78PdLdX/SKKOBrlQtPO4kjp0s5WRJI5iEIwL9b4efvAmF++CZiyF3kdNVqQBpoCtVi4rZogePx3A/elXnDIaJH0LTNvDyNfDJEzoBKQpooCtVC08szhYNRHoHuOV96HI5fPBbeHMiFBc5XZWqgQa6UrXwxOps0UAkueGaF+CS/4GvXofnLrOLe6mIFFCgi8gwEdksIrkiMtXP8etF5Evv41MR6RH8UpVyRmULPZaHLtZEBC76lV2p8dBOmDkYdn7idFXKj1oDXUTigenAcCAbGC8i2VVO2wEMMsZ0Bx4EZga7UKWcUhHojWKkS006DYNbF0OTFvDiaFjxjParR5hAWuh9gVxjzHZjTDEwGxjje4Ix5lNjzCHvp8uANsEtUynnNHHFk+qKb5xdLlVldoSJi+HcIbDgHnj7LrudpIoIgQR6a8C30yzP+7Xq3AK86++AiEwSkVUisurAgQOBV6mUwzxpSbGxyUUwJDeD8a/CwF/C5y/CrFFw7Funq1IEFuji52t+/84SkYuxgX6vv+PGmJnGmBxjTE5mZmbgVSrlMI+7EcwWrYu4eBhyH1wzC/ats/3qeaudrqrRCyTQ8wDf3SjaAHurniQi3YG/A2OMMbphoYopHrdLA92frlfaoY3xifD8cFjzitMVNWqBBPpKIEtEOoiICxgHzPM9QUTaAm8CPzHGbAl+mUo5K8OdRL52ufh3ZjeYuATaXgD/ug3+dQcUapeqE2oNdGNMKTAFWAhsBF4zxqwXkckiMtl72n1ABvCUiKwRkVUhq1gpB3jcSRwqKqa0TFch9Cs1A26Ya3c++nI2TOsFnz4JpfpLMJwC2uDCGLMAWFDlazN8Pr4VuDW4pSkVOTLdLoyBg0XFtExLdrqcyBSfAJfeDz2vh/f+H7z/G1j9vF3sq+NQp6trFHSmqFIBaPSTi+rCkwU3vA7XzbGfv3KNXQ8mf6uzdTUCGuhKBaBigS69MVoHHYfCbZ/B0Idg1zJ4qh8s/A2cPOJ0ZTFLA12pADTaBboaKsEFA6bAnauh53Xw2XR4ohesfgHKG8FyxGGmga5UADIa8wJdweBuCaOnwaQltkvm7bvs2PWvP3W4sNiiga5UANKSEnAlxOnQxYY6q6fd6u6qZ6GowI5df/2nuoJjkGigKxUAESFTZ4sGhwh0uxqmrIJBU2HTO/BkH7s5ta633iAa6EoFyM4W1RZ60LhS4OL/B1NW2pUclzwM0/vCujd1Fcd60kBXKkAedxL5x7SFHnTN29o1YW5+B5Kbw+sT4PkRsPUDKNeJXHWhga5UgDJ0PZfQav9D+K+PYNTjcHAbvHw1TDsfPvkbFB10urqooIGuVIA87iQKjhdTXq7dASETFw85E+Dn6+Dq56Bpa/jgPvhrZ5h7m13RUbtjqhXQ1H+llA30snLD4RMlpKe6nC4ntiW44AdX2ce+9bDyWfjyn7D2FWjVE/rcao+5UpyuNKJoC12pAFXMFi3QbpfwOqMrjHoU7t4II/4CpSdh3hR4tIudeVqwzekKI4YGulIB8ngnFzX6vUWdktwU+k6E25fZG6jnXgzLZ9iVHV+60g5/bOSzT7XLRakAZVZO/9ehi44SsTdQ2//Qbn33+Yuw6nmYfR00Oxt63wy9brSzUxsZbaErFaCMyhUXtYUeMdLOhEG/hp9/Bde+BOnnwL8fhEez4fVbYPtHUFbqdJVhoy10pQLUvEki8XGiQxcjUXwCZI+2jwNbYNWzdju8da/bTa3P+xF0HAbnDYGUdKerDRkNdKUCFBcnZKS6KNAul8iW2RGG/8luYp27GLYshK0LbbhLHJzdDzpeZgM+s5PtwokRGuhK1YFH13OJHq7U71rt5eWw9wvY8p59LPqdfbRob4O942XQ7kJISHK66gbRQFeqDjxpGuhRKS4O2vS2j0t+A0f22Fb7loWwepYdLeNy25EzHYdB1tCovKmqga5UHXjcLrbtL3S6DNVQzVpDzk/to7gIdi71tt4Xwsa3AYHWvb9rvZ/xA/tLIcJpoCtVBx53EgcKT2GMQWKo77VRc6V4+9Qvs8sK7FsHm71dMx8+BB/+wd5YPasXtMmxQd86B9yZTld+Gg10perA43ZRXFrOsVOlNE1OdLocFWwicGY3+xj0KyjcD7mLYPcK2LMKlj4Kxjt5qVlb24VTEfCteji+FIEGulJ1ULG3aEFhsQZ6Y+BuafdC7Xmd/by4CL5Za8N9z2q7WNj6ufaYxMMZ2d8FfOvedhRNXHzYytVAV6oOfDeL7uBJdbgaFXauFGjX3z4qFO73hrs35NfNtTdawd5oPet8G+5nnQ+ZnSG9Q8hG02igK1UHHp0tqqpyt4ROw+0D7BDJg9u+C/g9q+Cz6VBeYo9LHDRvZzfLzsgCz3nefzva52rAvRkNdKXqoGKBLh26qKoVF2fD2pMFPcfbr5WchP0boCAX8rdCwVbIz4UdS6H0xHffm9QUMs47PewzzoXEJrW+tAa6UnWQnupCBA7obFFVF4nJ0LqXffgqL4ejed6Q9wn7nZ/Y9d8riV14zJNV48tooCtVBwnxcbRI0a3oVJDExdk9VZu3tevM+Co+btd6r2jNF2y1gV8DDXSl6sjjdukmFyr0XKnQqrt9+JpcfR975E99UirC2PVctMtFRR4NdKXqSBfoUpEqoEAXkWEisllEckVkqp/jIiJPeI9/KSK9/D2PUrEgw+3SYYsqItUa6CISD0wHhgPZwHgRya5y2nAgy/uYBDwd5DqVihgedxLHi8s4Udy4969UkSeQm6J9gVxjzHYAEZkNjAE2+JwzBnjRGGOAZSLSXERaGWO+CXrFSjmsYm/RkU8sJT5OF+hSkSOQQG8N7Pb5PA+4IIBzWgPfC3QRmYRtwdO2bdu61qpURLioYyZXnt+aU6XaQlfht6iGY4EEur8miKnHORhjZgIzAXJyck47rlQ0OLNZMo/9uKfTZahG6ukbqj8WyE3RPOBsn8/bAHvrcY5SSqkQCiTQVwJZItJBRFzAOGBelXPmATd6R7v0A45o/7lSSoVXrV0uxphSEZkCLATigeeMMetFZLL3+AxgATACyAWKgAmhK1kppZQ/AU39N8YswIa279dm+HxsgDuCW5pSSqm60JmiSikVIzTQlVIqRmigK6VUjNBAV0qpGCH2fqYDLyxyDNjsyIvXjwfId7qIOtB6Q0vrDb1oqzlc9bYzxmT6O+DkBhebjTE5Dr5+nYjIKq03dLTe0Iq2eiH6ao6EerXLRSmlYoQGulJKxQgnA32mg69dH1pvaGm9oRVt9UL01ex4vY7dFFVKKRVc2uWilFIxQgNdKaViRMgDPZo2mBaRs0XkQxHZKCLrReRnfs4ZLCJHRGSN93GfE7X61LNTRL7y1rLKz/FIur6dfK7bGhE5KiI/r3KOo9dXRJ4Tkf0iss7na+ki8oGIbPX+26Ka763xvR7Geh8RkU3e/95zRaR5Nd9b43snjPXeLyJ7fP6bj6jme8N+fWuo+Z8+9e4UkTXVfG94r7ExJmQP7HK724BzABewFsiucs4I4F3srkf9gOWhrKmWelsBvbwfpwFb/NQ7GJjvVI1+at4JeGo4HjHX189741vsJImIub7ARUAvYJ3P1/4MTPV+PBX4UzU/T43v9TDWOxRI8H78J3/1BvLeCWO99wP3BPB+Cfv1ra7mKsf/CtwXCdc41C30yg2mjTHFQMUG074qN5g2xiwDmotIqxDX5Zcx5htjzOfej48BG7F7o0aziLm+VQwBthljvna6EF/GmI+Bg1W+PAZ4wfvxC8AVfr41kPd60Pmr1xjzvjGm1PvpMuwOYhGhmusbCEeuL9Rcs4gIcC3wajhqqU2oA726zaPrek7YiUh74HxguZ/D/UVkrYi8KyJdw1rY6Qzwvois9m7CXVVEXl/szlfV/U8QSdcX4Azj3YHL+29LP+dE6nX+KfYvNH9qe++E0xRvF9Fz1XRpRer1HQjsM8ZsreZ4WK9xqAM9aBtMh5OIuIE3gJ8bY45WOfw5tpugBzAN+Fe466viQmNML2A4cIeIXFTleCReXxcwGpjj53CkXd9AReJ1/g1QCrxczSm1vXfC5WngXKAn8A22C6OqiLu+XuOpuXUe1msc6kCPug2mRSQRG+YvG2PerHrcGHPUGFPo/XgBkCginjCX6VvPXu+/+4G52D9NfUXU9fUaDnxujNlX9UCkXV+vfRXdVN5/9/s5J6Kus4jcBIwCrjfeztyqAnjvhIUxZp8xpswYUw48U00dEXV9AUQkARgL/LO6c8J9jUMd6FG1wbS3P+xZYKMx5tFqzjnTex4i0hd7DQvCV+X3akkVkbSKj7E3w9ZVOS1irq+Pals1kXR9fcwDbvJ+fBPwlp9zAnmvh4WIDAPuBUYbY4qqOSeQ905YVLmnc2U1dUTM9fVxKbDJGJPn76Aj1zgMd4hHYEeLbAN+4/3aZGCy92MBpnuPfwXkhOuOsJ9af4j9M+5LYI33MaJKvVOA9di77MuAAQ7We463jrXemiL6+nrrScEGdDOfr0XM9cX+ovkGKMG2Cm8BMoDFwFbvv+nec88CFvh872nvdYfqzcX2N1e8h2dUrbe6945D9b7kfW9+iQ3pVpFyfaur2fv1WRXvW59zHb3GOvVfKaVihM4UVUqpGKGBrpRSMUIDXSmlYoQGulJKxQgNdKWUihEa6EopFSM00JVSKkb8f/6n9nuCYpNDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "## Compute the response for a given action and current temperature\n",
    "def respond(action, current_temp, tau):\n",
    "    return action + (current_temp - action) * math.exp(-1.0/tau)\n",
    "\n",
    "## Actions of a series of on, then off\n",
    "sAction = pd.Series(np.array([1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0]))\n",
    "sResponse = np.zeros(sAction.size)\n",
    "\n",
    "## Update the response with the response to the action\n",
    "for i in range(sAction.size):\n",
    "    ## Get last response\n",
    "    if i == 0:\n",
    "        last_response = 0\n",
    "    else:\n",
    "        last_response = sResponse[i - 1]\n",
    "    sResponse[i] = respond(sAction[i], last_response, 3.0)\n",
    "\n",
    "## Assemble and plot\n",
    "df = pd.DataFrame(list(zip(sAction, sResponse)), columns=['action', 'response'])\n",
    "df.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal and Reward\n",
    "The goal here is to make an agent that will take actions that will keep the temperature between 0.4 and 0.6.\n",
    "\n",
    "We make a reward function to reflect our goal.  When the temperature is between 0.4 and 0.6, we set the reward as 0.0.  When the temperature is outside of this band, we set the reward to be the negative distance the temperature is from its closest band.  So if the temperature is 0.1, then the reward is -(0.4 - 0.1) = -0.3, and if it is 0.8, then the reward is -(0.8 - 0.6) = -0.2.\n",
    "\n",
    "Let's chart the reward vs. temperature to show what is meant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Reward vs. Temperature')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtkAAAEWCAYAAACzG4tiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de7zcd13n8debFCVyMUADNIHQqjVRuQUrsJZdiYApFSEiKIpaUbfLCq52oUuLF0Ddbd14WxcQa5erCmGxhi4Xj72EAsqttbEFydEu96TQtBCgcFbS9LN/zC9wcpiZM+ecuc/r+Xicx5mZ329mvjO/md/vc77n8/l9UlVIkiRJ6p+7jHoAkiRJ0rQxyJYkSZL6zCBbkiRJ6jODbEmSJKnPDLIlSZKkPjPIliRJkvrMIFuSxkSS1yT5nVGPQ5K0dgbZkqZeko8nWUhye5LPNMHsPUY9rlFK8qzm/bi9eW/uXHT99lGPby2SnJXkplGPQ9JsM8iWNCt+pKruATwC2A5cOKqBJDlpVM99XFX9RVXdo3lPngQcOn69uW0sJblLkoEeu8Zh+0iafAbZkmZKVX0GmKMVbAOQ5JuT/F6STyb5bJJXJlnfLLsmyY81lx+bpJKc3Vx/QpL9zeVvT3J1ktuS3JrkL5JsWPQcH0/ywiQ3AF9OclKS7Un+IcmXkuwB7tZuzM34jiR5yKLbNjYz0PdLcnKStzbrfC7Ju/sRiCZ5UJK3NK/no0mes2jZxc1r3NPMfu9PclqSFzfrfzzJjkXrvy/Jbye5LskXkvxVkm9dtPzfJnl/8xr+IcmZS+77W0neD3wF2JTkPyQ50Lx3NyX5+Wbd+wJ/DXzbopn5+yZ5Y5JfX/SYJ8x2N//heEGSDwNfXO71S9JyDLIlzZQkD6Q1c7s4neB3ge+kFXh/B7AZ+M1m2TXA45rL/w74KPADi65fc/yhgYuATcB3AQ8CXrLk6X8S+GFgA639717g9cB9gP8N/Fi7MVfVvwKXNfc/7seBa6rqFuD5wKeBjcD9gRcB1fldWF6SdcDbgb9vXtNZwIuS/MCi1X4UeGXzeuaBq4EvAw8Afh94xZKH/VngWbTe329q1iHJqbTei1+j9V78OrA3yb0X3fenm/vfE/gMcDOt7Xgv4DnAy5N8T1Xd1ozro4tm5m/r8WX/BPBE4L49vn5J6sggW9Ks2JvkS8CngFuAFwMkCfDvgfOq6nNV9SXgvwHPbO53DScG1Rctuv4DzXKq6qaquqKq/rWqDgN/sGi94/64qj5VVQvAY4C7An9UVUer6s3AB7uM/y85Mcj+qeY2gKPAKcCDm8d6d1WtKcgGHgvcrap+t6q+WlX/DLyar78vAFdV1b6qugN4M62A9/eb628Eth3/j0Dj1VV1oKpup/X+H3895wCXVdWVVXVnVb0d+Cfghxbd99Kqmm9e3x1VdXlVfaxarqS1HR67xtf8h1V1qNk+vbx+SerIIFvSrNhVVfekNSu9DTi5uX0j8C3AdU2qwhHgb5rbAd4LfGeS+9Oa6X4d8KAkJwOPAt4F0KRtvDHJwSRfBP580XMc96lFlzcBB5cEw5/oMv6rgfVJHp3kwc1Y/rpZtpvWzPzfNmkNF/TwfiznwcCpx9+T5n35z7RmqY/77KLLC8DhRa9nofl990XrLH79nwC+pUkZeTDw00ue6wxa71G7+5LkKUk+0KTHHAF+kG98v1dq8XP08volqSOLOyTNlKq6JslrgN8DdgG30goIv6eqDrZZ/ytJrgN+BfhQVX01yd/TCrj+b1Xd2qx6Ea0UjYdV1W1JdgEvW/pwiy7fDGxOkkWB6Rbg/3YY951J3kRr9vezwFubWXea388Hnp/ke4B9ST5YVVet4K1Z6lPAgap66BoeY6kHLbq8BfhKVX0hyadozVT/cpf7fu29S3J3Wuk1TwfeUVV3JPkbWik7J6y7yJdp/TF1XLtgefH9BvH6Jc0QZ7IlzaI/Ap6Y5BFVdSfwZ8AfJrkfQJLNSXYuWv8a4Hl8Pf/6nUuuQytX+HbgSJLNwPnLjOG9wB3Af2qKIJ9Ga2a8m7+klTf8LL6eKkKSJyf5jib15YvAseZnLd7TPPavJrlbM8aHJXnkGh7z55J8Z1qnT3wJsKe5/bXAM5I8Psm6JOuby51mjdfTSrW5BbgzyVP4et48tP4IuV9OPE3jfuDJSTY026dbQA+Def2SZohBtqSZ0+RMvw74jeamF9JKt3hfk+pxJbB10V2uoRVEv6vDdYCXAo8EvgC8jVahYrcxfBV4GvBzwOdpBc/L3ef9tGZkNwHvWLTo9GbMt9MK3l9RVe8ESPKOJC/q9rgdnusocDbw/bRSOw4DfwKs5fR+rwfeABwE7qQ1+05VfZRW0edLaf1n4RO0/nPQ9hjV/PfgBcD/AW6j9R+Jty9a5R+By4FPNKke9wFeRWsbfxJ4azOOjgb0+iXNkKy9NkaSpO6SvA94WVX9+ajHIknD4Ey2JEmS1GcG2ZIkSVKfmS4iSZIk9Zkz2ZIkSVKfTeV5sk8++eQ69dRTRz0MSZIkTbHrrrvu1qra2G7ZVAbZp556Ktdee+2ohyFJkqQplqRjp17TRSRJkqQ+M8iWJEmS+swgW5IkSeozg2xJkiSpzwyyJUmSpD4zyJYkSZL6zCBbkiRJ6jODbEmSJKnPDLIlSZKkPjPIliRJkvrMIFuSJEnqs5EG2UnOSjKf5KYkF7RZniR/3Cy/IckjRzFOSZIkaSVOGtUTJ1kHvBx4IvBp4INJLq+qf1q02pOA05ufRwN/0vyWpIHae/1Bds/Nc+jIAps2rOf8nVvZtX1z12XTevusvmZJWotU1WieOPk3wEuqamdz/UKAqrpo0Tp/Cryzqt7QXJ8HHldVN3d77DPOOKOuvfbagY1d0nTbe/1BLrzsRhaOHvvabevvuo6LnvZQgLbLfux7N/NX1x2cuttn9TUbaEvqRZLrquqMtstGGGQ/HTirqn6xuf4zwKOr6nmL1nkrcHFVvae5fhXwwqrqGkEbZEtaizMvvpqDRxa+4fbNG9YDtF22LuFYm/3ppN8+q6/57y74wW+4XZKW6hZkjyxdBEib25bu7XpZp7Vici5wLsCWLVvWNjJJM+1Qm4Cy2+1A22BtGm73NUvS6owyyP408KBF1x8IHFrFOgBU1SXAJdCaye7fMCVNs3Y5uZs2rG87c7tpBmd1Z/U1m6staa1GeXaRDwKnJzktyTcBzwQuX7LO5cDPNmcZeQzwheXysSWpV8dzrw8eWaBoBZIXXnYjO7ZtZP1d152w7vq7ruP8nVs5f+fWtst+8tEPmsrbZ/E179i2se3nYu/1B5GkXo1sJruq7kjyPGAOWAe8qqo+nOQ5zfJXAm8HzgZuAr4CPHtU45U0fXbPzZ9Q9AawcPQY+w4c5qKnPbTrTGa7ZWc8+D5TefusveZOn4vdc/POZkvq2cgKHwfJwkdJvTjtgre1LfII8LGLf3jYw9GY8HMhqVfdCh/t+ChpZh3PN+71ds0GPxeS+sEgW9JM2Hv9Qc68+GpOu+BtnHnx1ey9/mDHXOPzd24d0Sg1Dpb7XLT7LEnSUgbZkqZepwJHgIue9lA2b1hPaJ0f2UYk2rV9c8fPRafPkoG2pKXMyZY09bo1l7HpiFbCz5KkxczJljTTVtNcRmrHz5KkXo2yGY0k9d1qmstIver2WbKBjaTFnMmWNDVW01xGWolORZE2sJG0lEG2pKmxXHMZCxy1Vp2KIvcdONyxgY2k2WS6iKSp0S1fdtf2zQbV6ot2n6Xz9uxvu6652tLsciZb0tSwiYhGxc+epKUMsiVNJJvLaJx0++zZvEaaTQbZkiaOzWU0bjrlagMWREozymY0kiaODUE0KfysStPNZjSSpooNQTQp/KxKs8sgW9LEschMk8LPqjS7DLIljTULHDXJLIiUZpdBtqSxZYGjJp0FkdLssvBR0tiyaEzTys+2NB0sfJQ0kSwa07Tysy1NP9uqSxoLe68/yO65eQ4dWWDThvWcv3MrmzasbzvbZ9GYJl23z3a774KpUNLkcSZb0sh1yr3esW2jBY6aSp0KInds22iutjQlDLIljdzuuXkWjh474baFo8fYd+CwBY6aSp0KIvcdONz2u7B7bn40A5W0aqaLSBq5bvmpu7ZvNqjWVGr32T5vz/6265qrLU0eZ7IljZwNO6QWvwvS9DDIljRUNpeROrN5jTQ9DLIlDY3NZaTubF4jTQ+b0UgaGhtwSKvjd0caTzajkTQWbMAhrY7fHWnyGGRLGhqLuqTV8bsjTR6DbEkDYYGj1D8WREqTxyBbUt9Z4Cj1lwWR0uSx8FFS31mkJQ2H3zVptCx8lDRUFmlJw+F3TRpftlWXtCZ7rz/I7rl5Dh1ZYNOG9Zy/cyubNqxvO7tmkZbUX92+a+2+m6ZmScPjTLakVeuUe71j20YLHKUh6FQQuWPbRnO1pREzyJa0arvn5lk4euyE2xaOHmPfgcMWOEpD0Kkgct+Bw22/m7vn5kczUGkGmS4iadW65YPu2r7ZoFoagnbftfP27G+7rrna0vA4ky1p1WyQIY0nv5vS6I0kyE5ynyRXJPmX5ve9O6z3qiS3JPnQsMco6UQ2l5Emh81rpNEb1Uz2BcBVVXU6cFVzvZ3XAGcNa1CS2rO5jDRZbF4jjd5ImtEkmQceV1U3JzkFeGdVtZ36SnIq8Naqekivj28zGqm/bHghTQe/y1J/jWMzmvtX1c0Aze/7rfUBk5yb5Nok1x4+fHjNA5T0dTa8kKaD32VpeAZ2dpEkVwIPaLPo1wbxfFV1CXAJtGayB/Ec0rTr1LzC5jLSdLB5jTQ8Awuyq+oJnZYl+WySUxali9wyqHFI6s3xvOvj59ZdnHd9/s6tJywDCxylSdTpu3y8eU2777+BtrQ6o0oXuRw4p7l8DvCWEY1DUqNTY5ndc/Mdi6g8+EqTxeY10vCMqhnNxcCbkvwC8EngGQBJNgGXVtXZzfU3AI8DTk7yaeDFVfW/RjNkabotl6tpcxlpOti8RhqOkQTZVXUb8Pg2tx8Czl50/SeHOS5plpl3Lc0uv/9S/9nxUZpBNpaRtJjNa6T+M8iWZoyNZSQtZfMaqf9G0oxm0GxGI3VmMwpJvXJ/IXU3js1oJI2IzSgk9cr9hbR6BtnSjOlUyGSBk6Sl3F9Iq2eQLU0xCxwlrcVy+wuLIqXODLKlKWWBo6S16taIqtM+xkBbarHwUZpSFixJGiT3MZKFj9JMsmBJ0iC5j5G6G1VbdUl9tPf6g+yem+fQkQU2bVjP+Tu32sFN0kB128e02yeZkqZZ40y2NOE65UXu2LbRAkdJA9OpKHLHto3maksYZEsTb/fcPAtHj51w28LRY+w7cNgCR0kD06koct+Bw233Sbvn5kczUGlETBeRJly3vMhd2zcbVEsamHb7mPP27G+7rrnamjXOZEsTzmYRksaJ+ySpxSBbmiA2l5E07rrtk2xeo1likC1NCJvLSJoEnXK1AQsiNVNsRiNNCBs/SJpk7sM0jWxGI00BGz9ImmTuwzRrDLKlCWExkaRJ5j5Ms8YgWxpDFjhKmjYWRGrWGGRLY8YCR0nTyIJIzRoLH6UxY3GQpFniPk+TzMJHaYJYHCRplrjP07Syrbo0QnuvP8juuXkOHVlg04b1nL9zK5s2rG87q2NxkKRp1G2f124faYqcJoUz2dKIdMq93rFtowWOkmZGp4LIHds2mqutidY1yE5yY5IbOv0Ma5DSNNo9N8/C0WMn3LZw9Bj7Dhy2wFHSzOhUELnvwOG2+8jdc/OjGai0Qsulizy5+f3c5vfrm9/PAr4ykBFJM6JbHuKu7ZsNqiXNjHb7vPP27G+7rrnamhRdZ7Kr6hNV9QngzKr6L1V1Y/NzAbBzOEOUppONGSSpM/eRmnS95mTfPcljj19J8v3A3QczJGn62FxGklbG5jWadL0G2T8PvDzJx5N8DHhFc5ukZdhcRpJWzuY1mnTLNqNJchfg6VX1piT3au7zhaGMbpVsRqNxYqMFSeof96kaJ2tqRlNVdwLPay5/cdwDbGnc2GhBkvrHfaomRa/NaK5I8gJgD/Dl4zdW1ecGMippQtlcRpIGy+Y1mhQrycl+LvAu4Lrmx3wMaRGby0jS4Nm8RpOipyC7qk5r8/Ntgx6cNElsLiNJg2fzGk2KXtNFSPIQ4LuBux2/rapeN4hBSZPI5jKSNBw2r9Ek6GkmO8mLgf/Z/OwA/jvwlAGOS5o4Nk6QpNFxH6xx02tO9tOBxwOfqapnAw8Hvnm1T5rkPkmuSPIvze97t1nnQUn2JflIkg8n+ZXVPp/UbzaXkaTxYvMajZteg+yF5lR+dzTnyr4FWEtO9gXAVVV1OnBVc32pO4DnV9V3AY8Bnpvku9fwnFJf2FxGksaPzWs0bpZtRgOQ5BXAi4BnAs8Hbgf2N7PaK3/SZB54XFXdnOQU4J1V1XW6L8lbgJdV1RXLPb7NaDRINkKQpMnhPluD1K0ZTU+Fj1X1S83FVyb5G+BeVXXDGsZ0/6q6uXnsm5Pcr9vKSU4FtgPv77LOucC5AFu2bFnD0KTubIQgSZPDfbZGpacgO8nrgHcD766qAz3e50rgAW0W/Vrvw4Mk9wD+CvjVqvpip/Wq6hLgEmjNZK/kOaSVsLmMJE0O99kalV5P4fca4LHA/0zybcB+4F1V9T863aGqntBpWZLPJjllUbrILR3WuyutAPsvquqyHscq9U277mHn79zKhZfdeML5WC1wlKTx1G2fbYdIDVKvzWiuBv4r8BvApcAZwH9cw/NeDpzTXD4HeMvSFZIE+F/AR6rqD9bwXNKqWOAoSZPPgkiNSq+Fj1cBdwfeSytt5D1V1Xb2uacnTe4LvAnYAnwSeEZVfS7JJuDSqjo7yWOb57oRuLO564uq6u3LPb6Fj+oHi2UkaXq5j1c/rLnwEbgB+F7gIcAXgCNJ3ltVq6oaqKrbaJ13e+nth4Czm8vvAbKax5f6wWIZSZpe7uM1aL2eXeQ8+FoR4rOBV9Mqalx1QxppXHTKybNYRpKmV7d9vLna6ode26o/L8keWgWPu4BXAU8a5MCkYeiUd20HR0mabp328Tu2bTRXW33Ra8fH9cAfANuq6vFV9dKmGFKaaLvn5k+oOAdYOHqM3XPzHYtlnM2QpMnXaR+/78DhjscFaSV6TRfZ3RQi/gzw6iQbgXtU1ccGOjppwJbLydu1fbNBtSRNqXb7+PP27G+7rrnaWqle00VeDLwQuLC56a7Anw9qUNKwdMqvNu9akmaTxwX1S6/pIj8KPAX4MnztLCD3HNSgpEHYe/1Bzrz4ak674G2cefHV5l1Lkr7BcseFdscSqZ1eg+yvVuuE2gWQ5O6DG5LUfzaWkST1ols9TrdieWmpXpvRvAA4HXgicBHw88AbquqPBzu81bEZjZay6YAkaa08lmipNTejqarfS/JE4IvAVuA3q+qKPo5RGiibDkiS1spjiVai146PNEH1FQBJ1iV5VlX9xcBGJvWRjWUkSWvlsUQr0TUnO8m9klyY5GVJfigtzwM+Cvz4cIYorYwFjpKkQeh2LLEgUkstV/j4elrpITcCvwj8LfAM4KlV9dQBj01aMQscJUmD0qkoErAgUt+ga+Fjkhur6qHN5XXArcCWqvrSkMa3KhY+zi6LUiRJw+axZ3Z1K3xcbib76PELVXUM+Ni4B9iabRalSJKGzWOP2lmu8PHhSb7YXA6wvrkeoKrqXgMdndTF3usPsntunkNHFti0YT3n79xqUYokaei6HXvaHatMVZwNXWeyq2pdVd2r+blnVZ206LIBtkamU+71jm0bLXCUJA1Vp4LIHds2mqs9w3rt+CiNld1z8ywcPXbCbQtHj7HvwGELHCVJQ9WpIHLfgcNtj1W75+ZHM1ANVc/nyZbGSbf8t13bNxtUS5KGqt2x57w9+9uua672bHAmWxOpU461udeSpHHhsWq2GWRr7NlcRpI0iWxeM9sMsjXWbC4jSZpUNq+ZbV2b0Uwqm9FMD0/wL0maNh7bpsdamtFII+UJ/iVJ08Zj22zw7CIaGzaXkSTNApvXzAZnsjUWbC4jSZoVNq+ZDQbZGgs2l5EkzQqb18wG00U0FmwuI0maJTavmX7OZGsseMJ+SdKs81g4XQyyNXQ2l5Ek6RvZvGa6GGRrqGwuI0lSezavmS42o9FQeQJ+SZJWxmPn+LIZjcaGJ+CXJGllPHZOJoNsDZVFHZIkrYzHzslkkK2BscBRkqS1syByMhlkayAscJQkqT8siJxMFj5qICzSkCRpsDzWjp6Fjxo6izQkSRosj7XjbSRt1ZPcB9gDnAp8HPjxqvr8knXuBrwL+GZa43xzVb14uCNVL/Zef5Ddc/McOrLApg3rOX/nVjZtWN/2r2uLNCRJ6o9ux9p2x2ZTM4drVDPZFwBXVdXpwFXN9aX+FfjBqno48AjgrCSPGeIY1YNOudc7tm20wFGSpAHqVBC5Y9tGc7XHwKiC7KcCr20uvxbYtXSFarm9uXrX5mf6Esgn3O65eRaOHjvhtoWjx9h34LAFjpIkDVCngsh9Bw63PTbvnpsfzUBn1EjSRYD7V9XNAFV1c5L7tVspyTrgOuA7gJdX1fs7PWCSc4FzAbZs2dL/Eautbvlgu7ZvNqiWJGmA2h1rz9uzv+265moP18BmspNcmeRDbX6e2utjVNWxqnoE8EDgUUke0mXdS6rqjKo6Y+PGjf14CeqBJ8iXJGm8eGweDwObya6qJ3RaluSzSU5pZrFPAW5Z5rGOJHkncBbwof6OVL1qV0Rx/s6tXHjZjSf8W8rca0mSRqfbsdmCyOEZVU725cA5zeVzgLcsXSHJxiQbmsvrgScAB4Y2Qp3A5jKSJE0Gm9eMh5E0o0lyX+BNwBbgk8AzqupzSTYBl1bV2UkeRqsoch2tPwbeVFW/1cvj24ym/zzhvSRJk81jef91a0YzksLHqroNeHyb2w8BZzeXbwC2D3lo6sAT3kuSNNk8lg+XHR/VE4soJEmabB7Lh8sgWyfYe/1Bzrz4ak674G2cefHVX8vT6nTCewscJUmaDN2O5Z2O/1q9UZ0nW2PoeHHj8WrkxcWNxwsZrUiWJGkydTqWA8se/7VyIyl8HDQLH1fHgghJkmaPx//V61b4aLqIvsaCCEmSZo/H/8EwXWRGtTsZ/aYN69v+JWtBhCRJ06vb8d/mNavnTPYM6tRYZse2jRY3SpI0YzoVRO7YttHmNWtgkD2Dds/Nn9BqFWDh6DH2HThs90ZJkmZMpw6R+w4cbhsv7J6bH81AJ4zpIjOoW+7Vru2bDaolSZox7Y7/5+3Z33Zdc7V740z2DPJk9JIkaTnGC2tjkD3l2p1c3sYykiRpOcvFCzaw6c4ge4p1KnAEzL2WJElddcrV3rV9c8cYw0D762xGM8U8ubwkSRoEY4wWm9HMKE8uL0mSBsEYY3meXWRK2FxGkiQNiw1sludM9hSwuYwkSRomG9gszyB7CthcRpIkDZMNbJZnusgUsLmMJEkaNhvYdOdM9hTwZPGSJGkcGJN8nUH2hLG5jCRJGlfdYpJZa15jkD1BbC4jSZLGWadcbWDmCiJtRjNBPPG7JEmaRNMaw9iMZkp44ndJkjSJZjGGMcieIBYTSJKkSTSLMYxB9piywFGSJE2LWSyINMgeQxY4SpKkaTKLBZEWPo6haS0OkCRJWmzSYx4LHyfMLBYHSJKk2TPNMY9t1Uds7/UH2T03z6EjC2zasJ7zd25l04b1bf+qm+biAEmSNHu6xTztYqRJSpF1JnuEOuVe79i20QJHSZI09ToVRO7YtnHic7UNskdo99w8C0ePnXDbwtFj7Dtw2AJHSZI09ToVRO47cLhtjLR7bn40A10F00VGqFse0q7tmw2qJUnS1GsX85y3Z3/bdScpV9uZ7BGaxROzS5IkLWcaYiSD7CGxuYwkSVJvpqF5jUH2ENhcRpIkqXfT0LzGZjRDMOknWpckSRoH4xZT2YxmxKb5ROuSJEnDMkkx1UiC7CT3SXJFkn9pft+7y7rrklyf5K3DHGM/TUPyviRJ0qhNUkw1qpnsC4Crqup04Krmeie/AnxkKKPqAwscJUmSBmOSYqpRBdlPBV7bXH4tsKvdSkkeCPwwcOmQxrUmFjhKkiQNTqeCyHGMqUZS+JjkSFVtWHT981X1DSkjSd4MXATcE3hBVT25l8cfVeHjuCXjS5IkaXC6FT4OrONjkiuBB7RZ9Gs93v/JwC1VdV2Sx/Ww/rnAuQBbtmxZwUj7Z5KS8SVJkjQ4Awuyq+oJnZYl+WySU6rq5iSnALe0We1M4ClJzgbuBtwryZ9X1U93eL5LgEugNZO99lewcps2rG87kz2OyfiSJEkanFHlZF8OnNNcPgd4y9IVqurCqnpgVZ0KPBO4ulOAPS4mKRlfkiRJgzOqIPti4IlJ/gV4YnOdJJuSvH1EY1qzSUrGlyRJ0uDY8VGSJElaBTs+SpIkSUNkkC1JkiT1mUG2JEmS1GcG2ZIkSVKfGWRLkiRJfWaQLUmSJPWZQbYkSZLUZwbZkiRJUp8ZZEuSJEl9ZpAtSZIk9ZlBtiRJktRnqapRj6HvkhwGPjHiYZwM3DriMWjw3M7Tz208G9zOs8HtPBuGuZ0fXFUb2y2YyiB7HCS5tqrOGPU4NFhu5+nnNp4NbufZ4HaeDeOynU0XkSRJkvrMIFuSJEnqM4Pswblk1APQULidp5/beDa4nWeD23k2jMV2NidbkiRJ6jNnsiVJkqQ+M8iWJEmS+swgew2SnJVkPslNSS5oszxJ/rhZfkOSR45inFqbHrbzs5rte0OSv0/y8FGMU2uz3HZetN73JTmW5OnDHJ/6o5ftnORxSfYn+XCSa4Y9Rq1dD/vtb03yf5L8Y7Odnz2KcWr1krwqyS1JPtRh+chjMIPsVUqyDng58CTgu4GfTPLdS1Z7EnB683Mu8CdDHaTWrMft/DHgB6rqYcBvMyYFF+pdj9v5+Hq/C8wNd4Tqh162c5INwCuAp1TV9wDPGPpAtSY9fp+fC/xTVT0ceBzw+0m+aagD1Vq9Bjiry/KRx2AG2av3KOCmqvpoVX0VeCPw1CXrPBV4XbW8D9iQ5JRhD1Rrsux2rqq/r6rPN4XQdsEAAAU9SURBVFffBzxwyGPU2vXyfQb4ZeCvgFuGOTj1TS/b+aeAy6rqkwBV5baePL1s5wLumSTAPYDPAXcMd5hai6p6F63t1snIYzCD7NXbDHxq0fVPN7etdB2Nt5Vuw18A3jHQEWkQlt3OSTYDPwq8cojjUn/18n3+TuDeSd6Z5LokPzu00alfetnOLwO+CzgE3Aj8SlXdOZzhaUhGHoOdNMwnmzJpc9vS8yH2so7GW8/bMMkOWkH2Ywc6Ig1CL9v5j4AXVtWx1uSXJlAv2/kk4HuBxwPrgfcmeV9V/fOgB6e+6WU77wT2Az8IfDtwRZJ3V9UXBz04Dc3IYzCD7NX7NPCgRdcfSOsv4pWuo/HW0zZM8jDgUuBJVXXbkMam/ullO58BvLEJsE8Gzk5yR1XtHc4Q1Qe97rdvraovA19O8i7g4YBB9uToZTs/G7i4Ws1CbkryMWAb8IHhDFFDMPIYzHSR1fsgcHqS05piiWcCly9Z53LgZ5sK18cAX6iqm4c9UK3Jsts5yRbgMuBnnO2aWMtu56o6rapOrapTgTcDv2SAPXF62W+/Bfi3SU5K8i3Ao4GPDHmcWptetvMnaf23giT3B7YCHx3qKDVoI4/BnMlepaq6I8nzaJ1lYB3wqqr6cJLnNMtfCbwdOBu4CfgKrb+cNUF63M6/CdwXeEUzy3lHVZ0xqjFr5XrczppwvWznqvpIkr8BbgDuBC6tqranCNN46vH7/NvAa5LcSCut4IVVdevIBq0VS/IGWmeGOTnJp4EXA3eF8YnBbKsuSZIk9ZnpIpIkSVKfGWRLkiRJfWaQLUmSJPWZQbYkSZLUZwbZkiRJUp95Cj9JGgNJ7gtc1Vx9AHAMONxcf1RVfXUkA+siyc8Db6+qz4x6LJI0bjyFnySNmSQvAW6vqt8bg7Gsq6pjHZa9B3heVe1fweOdVFV39G2AkjSmTBeRpDGX5JwkH0iyP8krktyl6Uh4JMnuJP+QZC7Jo5Nck+SjSc5u7vuLSf66WT6f5Nd7fNzfSfIB4FFJXprkg0k+lOSVTQe1nwAeAexp7v9NST6dZEPz2I9JcmVz+XeS/GmSK4BXN8/xB81z35DkF4f/rkrSYBlkS9IYS/IQ4EeB76+qR9BK83tms/hbgb+tqkcCXwVeQqtV9DOA31r0MI9q7vNI4KeSPKKHx/2HqnpUVb0X+B9V9X3AQ5tlZ1XVHmA/8BNV9Yge0lm2Az9SVT8DnAvcUlWPAr4PeG6SLat5fyRpXJmTLUnj7Qm0AtFrkwCsBz7VLFuoqiuayzcCX2haSt8InLroMeaq6vMASfYCj6W1/+/0uF8F/nrR/R+f5HzgbsDJwHXAO1b4Ot5SVf+vufxDwHclWRzUnw58coWPKUljyyBbksZbgFdV1W+ccGNyEq1g+Lg7gX9ddHnx/n1p8U0t87gL1RTsJPkW4GXAI6vqYJLfoRVst3MHX/8P6dJ1vrzkNf1SVV2FJE0p00UkabxdCfx4kpOhdRaSVaRW/FCSDU3A/FTg71bwuOtpBe23Jrkn8GOLln0JuOei6x8Hvre5vHi9peaAX2oCepJsTbJ+ha9JksaaM9mSNMaq6sYkLwWuTHIX4CjwHODQCh7mPcBfAt8OvP742UB6edyqui3Ja4EPAZ8A3r9o8auBS5Ms0Mr7fgnwZ0k+A3ygy3j+FNgC7G9SVW6hFfxL0tTwFH6SNMWaM3c8pKp+ddRjkaRZYrqIJEmS1GfOZEuSJEl95ky2JEmS1GcG2ZIkSVKfGWRLkiRJfWaQLUmSJPWZQbYkSZLUZ/8fr4DaiEXcvH8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def reward(temp):\n",
    "        delta = abs(temp - 0.5)\n",
    "        if delta < 0.1:\n",
    "            return 0.0\n",
    "        else:\n",
    "            return -delta + 0.1\n",
    "\n",
    "temps = [x * 0.01 for x in range(100)]\n",
    "rewards = [reward(x) for x in temps]\n",
    "\n",
    "fig=plt.figure(figsize=(12, 4))\n",
    "\n",
    "plt.scatter(temps, rewards)\n",
    "plt.xlabel('Temperature')\n",
    "plt.ylabel('Reward')\n",
    "plt.title('Reward vs. Temperature')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Environment Setup\n",
    "\n",
    "The environment responds to actions.  It is what keeps track of the temperature state of the room, returns the reward for being in that temperature state, and tells you if the episode is over or not (in this case, we just set a max episode length that can happen).\n",
    "\n",
    "Here is the gist of the flow:\n",
    "\n",
    "- Create an environment by calling `Environment.create()`, see below, telling it to use the class you created for this (here, the ThermostatEnvironment) and the max timesteps per episode.  The enviroment is assigned to the name `environment`.\n",
    "- Initialize the environment `environment` by calling `environment.reset()`.  This will do stuff, most importantly, it will initialize the `timestep` attribute to 0.\n",
    "- When you want to take an action on the current state of the environment, you will call `environment.execute(<action-value>)`.  If you want to have the heater off, you call `environment.execute(0)`, and if you want to have the heater on, you call `environment.execute(1)`.\n",
    "- What the `execute()` call returns is a tuple with 3 entries:\n",
    "    - __state__.  In this case, the state is the current temperature that results from taking the action.  If you turn on the heater, the temperature will rise from the previous state, and if the heater was turned off, the temperature will fall from the previous state.  This should be kept as a numpy array, even though it seems like overkill with a single value for the state coming back.  For more complex examples beyond this thermostat, there will be more than 1 component to the state.\n",
    "    - __terminal__.  This is a True/False value.  It is True if the episode terminated.  In this case, that will happen once you exceed the max number of steps you have set.  Otherwise, it will be False, which lets the agent know that it can take further steps.\n",
    "    - __reward__.  This is the reward for taking the action you took.\n",
    "\n",
    "Below, to train the agent, you will have the agent take actions on the environment, and the environment will return these signals so that the agent can self-train to optimize its reward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "###-----------------------------------------------------------------------------\n",
    "## Imports\n",
    "from tensorforce.environments import Environment\n",
    "from tensorforce.agents import Agent\n",
    "\n",
    "\n",
    "\n",
    "###-----------------------------------------------------------------------------\n",
    "### Environment definition\n",
    "class ThermostatEnvironment(Environment):\n",
    "    \"\"\"This class defines a simple thermostat environment.  It is a room with\n",
    "    a heater, and when the heater is on, the room temperature will approach\n",
    "    the max heater temperature (usually 1.0), and when off, the room will\n",
    "    decay to a temperature of 0.0.  The exponential constant that determines\n",
    "    how fast it approaches these temperatures over timesteps is tau.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        ## Some initializations.  Will eventually parameterize this in the constructor.\n",
    "        self.tau = 3.0\n",
    "        self.current_temp = np.random.random(size=(1,))\n",
    "\n",
    "        super().__init__()\n",
    "\n",
    "\n",
    "    def states(self):\n",
    "        return dict(type='float', shape=(1,))\n",
    "\n",
    "\n",
    "    def actions(self):\n",
    "        \"\"\"Action 0 means no heater, temperature approaches 0.0.  Action 1 means\n",
    "        the heater is on and the room temperature approaches 1.0.\n",
    "        \"\"\"\n",
    "        return dict(type='int', num_values=2)\n",
    "\n",
    "\n",
    "    # Optional, should only be defined if environment has a natural maximum\n",
    "    # episode length\n",
    "    def max_episode_timesteps(self):\n",
    "        return super().max_episode_timesteps()\n",
    "\n",
    "\n",
    "    # Optional\n",
    "    def close(self):\n",
    "        super().close()\n",
    "\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"Reset state.\n",
    "        \"\"\"\n",
    "        # state = np.random.random(size=(1,))\n",
    "        self.timestep = 0\n",
    "        self.current_temp = np.random.random(size=(1,))\n",
    "        return self.current_temp\n",
    "\n",
    "\n",
    "    def response(self, action):\n",
    "        \"\"\"Respond to an action.  When the action is 1, the temperature\n",
    "        exponentially decays approaches 1.0.  When the action is 0,\n",
    "        the current temperature decays towards 0.0.\n",
    "        \"\"\"\n",
    "        return action + (self.current_temp - action) * math.exp(-1.0 / self.tau)\n",
    "\n",
    "\n",
    "    def reward_compute(self):\n",
    "        \"\"\" The reward here is 0 if the current temp is between 0.4 and 0.6,\n",
    "        else it is distance the temp is away from the 0.4 or 0.6 boundary.\n",
    "        \n",
    "        Return the value within the numpy array, not the numpy array.\n",
    "        \"\"\"\n",
    "        delta = abs(self.current_temp - 0.5)\n",
    "        if delta < 0.1:\n",
    "            return 0.0\n",
    "        else:\n",
    "            return -delta[0] + 0.1\n",
    "\n",
    "\n",
    "    def execute(self, actions):\n",
    "        ## Check the action is either 0 or 1 -- heater on or off.\n",
    "        assert actions == 0 or actions == 1\n",
    "\n",
    "        ## Increment timestamp\n",
    "        self.timestep += 1\n",
    "        \n",
    "        ## Update the current_temp\n",
    "        self.current_temp = self.response(actions)\n",
    "        \n",
    "        ## Compute the reward\n",
    "        reward = self.reward_compute()\n",
    "\n",
    "        ## The only way to go terminal is to exceed max_episode_timestamp.\n",
    "        ## terminal == False means episode is not done\n",
    "        ## terminal == True means it is done.\n",
    "        terminal = False\n",
    "        if self.timestep > self.max_episode_timesteps():\n",
    "            terminal = True\n",
    "        \n",
    "        return self.current_temp, terminal, reward\n",
    "\n",
    "###-----------------------------------------------------------------------------\n",
    "### Create the environment\n",
    "###   - Tell it the environment class\n",
    "###   - Set the max timestamps that can happen per episode\n",
    "environment = environment = Environment.create(\n",
    "    environment=ThermostatEnvironment,\n",
    "    max_episode_timesteps=100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Agent setup\n",
    "\n",
    "Here we configure a type of agent to learn against this environment.  There are many agent configurations to choose from, which we will not cover here.  We will not discuss what type of agent to choose here -- we will just take a basic agent to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\matth\\Anaconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\ops\\resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    }
   ],
   "source": [
    "agent = Agent.create(\n",
    "    agent='tensorforce', environment=environment, update=64,\n",
    "    objective='policy_gradient', reward_estimation=dict(horizon=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check: Untrained Agent Performance\n",
    "\n",
    "Let's see how the untrained agent performs on the environment.  The red horizontal lines are the target bands for the temperature.\n",
    "\n",
    "The agent doesn't take actions to try and get the temperature within the bands.  It either initializes a policy to the heater always off or always on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAEWCAYAAABPDqCoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZxcZZ3v8c+vl+wrWSR0EggYDBgWISIgRFRUcEZxHFFwX5F7RdHRK3jdtxncgQFFBnH0DoqIjjAKsgkiKJiALEkIEIOQFRICSci+/O4fdYJF00mqSFdXdfXn/XrVq+osdc6v6oHOt59+znMiM5EkSZJUmZZ6FyBJkiT1JgZoSZIkqQoGaEmSJKkKBmhJkiSpCgZoSZIkqQoGaEmSJKkKBmhJUs1FxO8i4i31rkOSuoMBWlKvEBFPlT22RsS6suW31bu+XRERSyPiqHrX8VxFxLFlbbEmIrJTe43NzFdk5s96uK5e/b1Kalxt9S5AkiqRmUO2vY6IvwHvz8zr61dRZSKiLTM39/Zz7EjRDkOKWqYAs8rbS5KajT3QkppCRLRGxGcjYn5ELI+ISyJiRLFtSkRsjoj3RcSiiHg8It4bEUdExKyIeDIivl12rFOLIQffj4hVETEnIqaXbd8tIn5c9HAuiIjPR0RLp/eeHxFPAGcW578pIlZExLKI+FFEDC32/zkwFri26K39SEQcFxHzOn2+p3tTI+KsiPhJRPwsIlYDJ+3o83fxXc2PiGPLlvtHxMqI2D8iBkfEpUWtT0bE7RExshva57aIeHun7+i84rwPRsS0iDilaJ9HI+KksvcOjIizi+96aUT8e0T0L7btHhG/LWp9PCJ+t73vtVh/dPGZnoyIOyPipZ1q/HJE3FHU9YuIGL6rn11S8zFAS2oW/wd4NXAUMB7YBHynbHsrcCCwN/Ae4N+BTwAvK9a/JyJeUrb/dOBuYBRwFvCriBhWbLsEWFkc6zDgDcA7Or33LmA08K1i3ZeA3YEDgBcAnwbIzBOBx4BXZ+aQzDy3ws/7z8CPgOHALyr4/OUuBU4uW/5H4KHMnAO8n9JfJzuK+k8DNlZYUzWOBv5I6fv9FaXPsB8wCfgA8L2IGFDs+x1Kn2nbd7cvcGax7Qzg/qLWccAXoOvvNSL2Ks71aWA34DOU2rX8F4R3Am+j9Pn78ff2k6SnGaAlNYsPAmdm5uLMXA98EXhLRETZPl/KzA2ZeWWx/OPMfDwzH6EU5l5Utu+CzPxuZm7KzB8DC4HXRMSelALyv2Tm2sxcApwLnFT23vmZ+R+ZuSUz12Xm3Mz8XWZuzMylwNmUgvuu+H1mXpWZWzNzXYWff5tLgDdGRL9i+a3AT4rXm4AxwD6ZuTkzZ2Tmml2stStzM/MnxdCTy4CJwBeK7+hKSuF1r4hoA94LnJ6ZT2bmSkq/0Gz7vjcBewATi/fevINzvgv4ZWZeX3xvVwFzKP3isc0Pi/Z6Cvg8z/xFQ5IAx0BLagJFSJwAXBURWbaphVIPJ8CWzHy8bNs64NFOy+Xjdhd2Os3DlILansAAYFlZNm0ByodcLOhU3x7AOcCRwNBi/yWVfLYdePocFXz+5eVvzMzZEbEAOD4ibgSOBz5abP4BpZ7yyyNiCPBj4LOZuWUX6+2s83e/oQjH5euGUPrO24HZZd93ANvGfH+VUu/+jRGxCfhuZn6bru0JnBwRJ5atay/OsU152z0MDIqI4Z1qk9THGaAl9XqZmRGxCHhjZt7ReXtEjH4Ohx3faXkisJhSwHoKGJmZ+ax3FSV1Wv4GsAaYmplPFON7v7KD/dcAg7YtREQ7pSEHXZ5jZ59/O35KqXd1N2BGZi4ojrUB+BzwuYjYG7gGmE2p17oellAKy/t0+gUIgCLYng6cHhEHUQrSt2fmrTz7e10AXJSZH97B+SaUvZ4IrDU8S+rMIRySmsUFwFkRMQEgIsZGxOt24XgTiovd2oqL3yYC12bmQ8BtwNcjYmhEtETE5NjxdGlDKYXuVRExEfiXTtsfpTSeepv7gN0i4pVFeP4iO/95Xe3n/ymlsc/v5+/DN7ZNSbd/lC6KXEUpvHZ373PFMnMTcDFwTkSMjpIJEfGqot7XR8Skohd+ZVHrtno7f68/Ak4svtfW4uLEV0bE7mX7vDsi9i16378A9OjUe5J6BwO0pGbxdeB64HdRmpnij8Ahu3C8mymNiV5B6aKzfyrriTwZGAHMLbb/DHjeDo71OUoX960E/pvSBXPlvgp8tZgZ4rTMXE6pV/USSkNJltJpGEYXqvr8mfk3ShdJvhj4edmmDuAKYDUwC7iK0hhlIuI/I+LsndRRCx+l1Ps/k9J3+Fvg+cW2/YCbKNV7M/DNzLyt2Nb5e51P6eLLL1L6Ph+m9D2X/1v4/yj9crEI2Ap8vHYfS1JvFdv/C6Qk9U0RcSrwpsw8dqc7q2lExG3AeZn5X/WuRVJjswdakiRJqkLNAnREXBwRj0XErO1sj4g4NyLmRcQ9EbErf2qVJEmSekTNhnBE6a5dT1GaZ3VqF9tfC3wYeC3wEuCczHxJ5/0kSZKkRlKzHuhiMvsVO9jlBErhOosLPkZExLha1SNJkiR1h3rOA93BMyesX1ise9bNBSLiFOAUgMGDBx86ZcqUHilQkiRJfdcdd9yxPDPHdF5fzwDd1e1luxxPkpkXAhcCTJs2LWfOnFnLuiRJkiQi4uGu1tdzFo6FPPOOT+MpzfMpSZIkNax6BugrgXcWs3EcDqzMzGcN35AkSZIaSc2GcETET4FjgNERsRD4PNAOkJkXULq71WuBecBa4D21qkWSJEnqLjUL0Jl58k62J/ChWp1fkiRJqgXvRChJkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVcEALUmSJFXBAC1JkiRVwQAtSZIkVaGmAToijouI+yNiXkSc2cX24RHxPxFxd0TMjoj31LIeSZIkaVfVLEBHRCtwPnA8sD9wckTs32m3DwFzMvMg4BjgWxHRr1Y1SZIkSbuqlj3QhwHzMnN+Zm4ELgVO6LRPAkMjIoAhwApgcw1rkiRJknZJLQN0B7CgbHlhsa7cecB+wGLgXuD0zNza+UARcUpEzIyImcuWLatVvZIkSdJO1TJARxfrstPya4C7gD2Ag4HzImLYs96UeWFmTsvMaWPGjOn+SiVJkqQK1TJALwQmlC2Pp9TTXO49wC+zZB7wEDClhjVJkiRJu6SWAXoGMDkiJhUXBp4EXNlpn0eAVwJExPOAFwDza1iTJEmStEvaanXgzNwcEacB1wCtwMWZOTsiTi22XwB8GfjPiLiX0pCPMzJzea1qkiRJknZVzQI0QGZeBVzVad0FZa8XA6+uZQ2SJElSd/JOhJIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhUM0JIkSVIVKgrQETE+Il5evO4fEYNrW5YkSZLUmHYaoCPivcCVwEXFqj2BK2pZlCRJktSoKumB/ghwOLAKIDMfAMbWsihJkiSpUVUSoNdn5sZtCxHRCkTtSpIkSZIaVyUB+taI+CQwoBgH/TPg17UtS5IkSWpMlQToTwKrgbnA6cANwKdrWZQkSZLUqNp2tLEYrnFxZr4L+F61B4+I44BzgFbgosw8q4t9jgHOBtqB5Zn5smrPI0mSJPWUHQbozNwSEeMioj0zN1Vz4CJ8nw+8ClgIzIiIKzNzTtk+I4DvAsdl5iMR4cWJkiRJamg7DNCF+cAfIuIKYM22lZl57k7edxgwLzPnA0TEpcAJwJyyfd4K/DIzHymO+VgVtfe8Y46pdwWSJEnN6aab6l1BxSoZA70MuA4YBIwpe+xMB7CgbHlhsa7cvsDIiLgpIu6IiHd2daCIOCUiZkbEzGXLllVwakmSJKk2dtoDnZmffY7H7mqqu+zi/IcCrwQGAn+KiNuKuabLa7gQuBBg2rRpnY/Rc3rRb0aSJEmqjZ0G6Ii4jmcHXzLz1Tt560JgQtnyeGBxF/ssz8w1wJqIuBk4CHgASZIkqQFVMgb6M2WvBwD/DGyo4H0zgMkRMQlYBJxEacxzuSuA8yKiDegHvAT4TgXHliRJkuqikiEct3da9fuI+H0F79scEacB11Caxu7izJwdEacW2y/IzPsi4rfAPcBWSlPdzar6U0iSJEk9pJIhHMPKFlsojVkeV8nBM/Mq4KpO6y7otPwN4BuVHE+SJEmqt0qGcMymNAY6gM3AQ8AHalmUJEmS1KgqCdB7d76JSjFmWZIkSepzKpkHuvMYaIA/d3chkiRJUm+w3Z7k4rba44CBEXEAf5/XeRilm6pIkiRJfc6OhmL8A/BeSvM3f7ds/Wrgud5cRZIkSerVthugM/OHwA8j4s2ZeVkP1iRJkiQ1rErmgb4sIl4DvJDSjVS2rf/XWhYmSZIkNaJK5oH+LjACmA78kNKdCG+rcV2SJElSQ6pkFo6jMvOtwOOZ+VlKt9seX9uyJEmSpMZUSYBev+05InYvlveqWUWSJElSA6vkhihXRcQI4JvAXcAW4Ec1rUqSJElqUDsM0BHRAlydmU8CP4+IXwMDM3NFj1QnSZIkNZgdDuHIzK3AOWXL6wzPkiRJ6ssqGQN9XUScUPNKJEmSpF6gkjHQpwHDI2IDsI7SLb0zM3eraWWSJElSA6qkB3o00A4MAcYUy2NqWVSjWbJyHR/6yZ3cPv/xepciSZKkOttpgM7MLcCJwBnF63HAwbUurJEM7t/Gb+5Zwp8fcvi3JElSX7fTAB0R5wEvB95RrFoLXFDLohrNsAHtTBo9mFmLV9a7FEmSJNVZJUM4jszMD1LcUKWYhaNfTatqQFM7hjNr0ap6lyFJkqQ6qyRAbyrmg06AiBgFbK1pVQ1o6h7DWPTkOlas2VjvUiRJklRHlQTo84FfAGMi4ovALcDXalpVAzqgYzgAsxY5jEOSJKkv2+k0dpn544i4Azi2WHViZs6qbVmN54V7FAF68Uqm79unJiGRJElSmUrmgQZoBTZRGsZRSa910xk+qJ2Juw2yB1qSJKmPq2QWjk8DPwX2AMYDP4mIT9W6sEY0tWOYFxJKkiT1cZX0Jr8deHFmfiYzPw0cBryztmU1pqkdw3lkxVpWrt1U71IkSZJUJ5UE6Id55lCPNmB+bcppbFPLxkFLkiSpb6okQK8FZkfERRHxH8C9wJMR8e2I+HZty2sszsQhSZKkSi4i/E3x2Oa2GtXS8EYO7kfHiIHca4CWJEnqsyqZxu4HPVFIbzG1YxizF3shoSRJUl9VySwcx0XEjIh4LCJWRMQTEbGiJ4prRAd0DOeh5WtYtd4LCSVJkvqiSsZAnwd8EOgAxgCji+c+6YXFOOg59kJLkiT1SZUE6IXAXZm5KTO3bHvUurBG9fRMHI6DliRJ6pMquYjwk8D/RMRNwIZtKzPz3FoV1cjGDO3P7sMGGKAlSZL6qEoC9Bcp3cZ7BLC1tuX0DlM7hjsThyRJUh9VSYAem5mH1rySXuSAjuHcMPdRntqwmSH9K/kKJUmS1CwqGQN9Q0S84rkcvJjB4/6ImBcRZ+5gvxdHxJaIeNNzOU9Pm9oxjEy4b4kXEkqSJPU1lQToDwDXR8RT1UxjFxGtwPnA8cD+wMkRsf929vsacE11pdfPtjsS3rvQYRySJEl9TSXjD0Y/x2MfBszLzPkAEXEpcAIwp9N+HwZ+Abz4OZ6n5xxzDABjgTGH/C9m/fAy+MzVdS1JkiSpKdx0U70rqNhOe6CLKetOBM4oXo8DDq7g2B3AgrLlhcW6p0VEB/BPwAU7OlBEnBIRMyNi5rJlyyo4de0dsOZRZg1+Xr3LkCRJUg/baQ90RJwHtAPTgX8F1lIKvDvrMY4u1mWn5bMpgnlEV7sXb8q8ELgQYNq0aZ2P0XPKfjOaeu393HTjPNZdewMD+7XWrSRJkiT1rEqGcByZmYdExF8AMnNFRPSr4H0LgQlly+OBxZ32mQZcWoTn0cBrI2JzZv6qguPX1dSO4WxNmLNkFYfuObLe5UiSJKmHVHIR4aaIaKHoPY6IUVQ2H/QMYHJETCoC90nAleU7ZOakzNwrM/cCLgf+d28IzwAHjPeOhJIkSX3RdgN0RGzrnT6f0kV+YyLii8AtlGbN2KHM3AycRml2jfuAyzJzdkScGhGn7nLldbb7sAGMGtzPAC1JktTH7GgIx5+BQzLzxxFxB3AspXHNJ2bmrEoOnplXAVd1WtflBYOZ+e6KKm4QEeEdCSVJkvqgHQXop6/qy8zZwOzal9O7TO0Yxi3zlrN+0xYGtHshoSRJUl+wowA9JiL+ZXsbM/PbNainVzmgYzhbtiZzl67m4Akj6l2OJEmSesCOLiJsBYYAQ7fz6PNeuMe2OxI+WedKJEmS1FN21AO9JDO/1GOV9ELjRw5k92EDuG3+Ct5xxF71LkeSJEk9YEc90Nu/s4mA0oWER08ezS3zlrNla/3u7yJJkqSes6MA/coeq6IXm77vGFau28TdDuOQJEnqE7YboDNzRU8W0lsd9fzRRMAfHlhe71IkSZLUAyq5E6F2YOTgfhzYMZybH1xW71IkSZLUAwzQ3eDoyWO4a8GTrFy3qd6lSJIkqcYM0N1g+r5j2LI1+dNfHcYhSZLU7AzQ3eBFE0cwpH8bNz9ogJYkSWp2Buhu0N7awhH7jOLmB5aR6XR2kiRJzcwA3U2m7zuGhU+s46Hla+pdiiRJkmrIAN1Npk8eDcAfHMYhSZLU1AzQ3WTPUYPZc9Qgbn7A6ewkSZKamQG6Gx09eTR/mv84GzdvrXcpkiRJqhEDdDeaPnkMazdu4Y6Hn6h3KZIkSaoRA3Q3OmKfUbS1BH/wroSSJElNywDdjYYOaOeQiSO9rbckSVITM0B3s6Mnj2bWolUsf2pDvUuRJElSDRigu9n0fccAcOs8p7OTJElqRgbobja1YzgjB7Xze6ezkyRJakoG6G7W2hK89Pmj+cODy72ttyRJUhMyQNfA9H3HsGz1BuYuXV3vUiRJktTNDNA1cHRxW2+HcUiSJDUfA3QNjBs+kAM6hvOrvyxyGIckSVKTMUDXyJunjWfu0tXMWrSq3qVIkiSpGxmga+T1B3fQv62Fy2YuqHcpkiRJ6kYG6BoZPrCd46buzq/uWsT6TVvqXY4kSZK6iQG6ht4ybQKr12/mmtlL612KJEmSuokBuoYO33sUE3YbyM9mOIxDkiSpWRiga6ilJTjx0An88a+Ps2DF2nqXI0mSpG5ggK6xNx06ngj4uRcTSpIkNQUDdI3tMWIgR08ew+V3LGTLVueEliRJ6u1qGqAj4riIuD8i5kXEmV1sf1tE3FM8/hgRB9Wynnp5y7QJLF65nlvmLa93KZIkSdpFNQvQEdEKnA8cD+wPnBwR+3fa7SHgZZl5IPBl4MJa1VNPx+4/lpGD2p0TWpIkqQnUsgf6MGBeZs7PzI3ApcAJ5Ttk5h8z84li8TZgfA3rqZv+ba284UUdXDf7UZ5Ys7He5UiSJGkX1DJAdwDlXa4Li3Xb8z7g6q42RMQpETEzImYuW7asG0vsOW+eNoGNW7byq7sW1bsUSZIk7YJaBujoYl2XV9FFxMspBegzutqemRdm5rTMnDZmzJhuLLHn7DduGAeOH87PZiwg04sJJUmSeqtaBuiFwISy5fHA4s47RcSBwEXACZn5eA3rqbsTp01g7tLVzFq0qt6lSJIk6TmqZYCeAUyOiEkR0Q84CbiyfIeImAj8EnhHZj5Qw1oawusP2oP+bS1ccvvD9S5FkiRJz1HNAnRmbgZOA64B7gMuy8zZEXFqRJxa7PY5YBTw3Yi4KyJm1qqeRjB8YDtvnjaBy+9YyEPL19S7HEmSJD0H0dvG406bNi1nzuy9Ofux1et52ddv4hX7jeX8tx5S73IkSZK0HRFxR2ZO67zeOxH2sLFDB/CBoyfxm3uWcM/CJ+tdjiRJkqpkgK6DD0zfm90G9+Nrv51b71IkSZJUJQN0HQwd0M5pL38+t857nD882DvntZYkSeqrDNB18rbDJzJ+5EDOunouW7f2rnHokiRJfZkBuk76t7Xy8Vfvy+zFq/j1vUvqXY4kSZIqZICuoxMO6mDK7kP55jX3s3Hz1nqXI0mSpAoYoOuopSU447gpPLJiLZfOeKTe5UiSJKkCBug6O+YFY3jJpN0494YHWbNhc73LkSRJ0k4YoOssIjjj+Cksf2ojF948v97lSJIkaScM0A3gkIkjed1Be3D+jfP4yyNP1LscSZIk7YABukF85YSp7D58AKf95C+sXLup3uVIkiRpOwzQDWL4oHbOe+shPLZ6PZ+4/G4ynRtakiSpERmgG8jBE0Zw5vH7cd2cR7n41r/VuxxJkiR1wQDdYN770r141f7P46yr7+OuBU/WuxxJkiR1YoBuMBHBN990EGOHDuBDl9zpeGhJkqQGY4BuQMMHtXP+2xwPLUmS1IgM0A3q4AkjOOO4KVw351F+cMtD9S5HkiRJhbZ6F6Dte99Rk5jxtxV85Tf30b+thXccsVe9S5IkSerz7IFuYBHBOSe9iGP3G8tnr5jNhTf/td4lSZIk9XkG6AY3oL2V7739UP7hwHH861VzOfv6BxwTLUmSVEcO4egF2ltbOPekFzGwvZWzr3+QdRu3cObxU4iIepcmSZLU5xige4nWluDr/3wgA9tb+f7N81m3aQtfeN0LaWkxREuSJPUkA3Qv0tISfOmEFzKwXysX3jyf1es385U3TGVwf5tRkiSpp5i8epmI4FPHT2HYgDa+dd0D3PnIE3zrxIOYttdu9S5NkiSpT/Aiwl4oIjjtFZO59AOHszWTE7//J/7t6vvYsHlLvUuTJElqegboXuwle4/i6tOnc9KLJ/L938/nhPNuZfbilfUuS5IkqakZoHu5If3b+Lc3HsAP3/1iVqzZyBvOv5Vzrn+QNRs217s0SZKkpmSAbhIvnzKWaz82neOmjuM71z/A0V+/kfNvnMfq9ZvqXZokSVJTid52U45p06blzJkz611GQ7vzkSf49xse5Mb7lzF8YDvvO2oS7zpyL4YPbK93aZIkSb1GRNyRmdOetd4A3bzuXbiSc3/3INfNeZSh/dt455F78qZDJzBp9OB6lyZJktTwDNB92JzFqzjvxge5etZSMuGg8cN5/cEdvO7AcYwdNqDe5UmSJDUkA7RYsnIdv757CVfcvYhZi1bREnDEPqN43YF7cPS+Y+gYMbDeJUqSJDUMA7SeYd5jT3Hl3Yu54q5FPPz4WgAm7jaIw/fejSP2GcXhe49i3HADtSRJ6rsM0OpSZjJ36Wpum/84f/rr49z+0ApWrivN3LHnqEFM3WM4U3Yfyn7jhjFl3FA6RgwkIupctSRJUu0ZoFWRrVuT+5au4k9/fZwZf1vBfUtW88iKtU9vHzqgjSm7D2XPUYOZMHIQE3YbyITdBjFh5CDGDu1PS4vhWpIkNYe6BOiIOA44B2gFLsrMszptj2L7a4G1wLsz884dHdMA3fOe2rCZ+5euZu7SVcxdspr7l67m4RVreHTVhmfs16+thbFD+zNmaH9GDyk9jxnSn9FD+zNyUDvDBz7zMXRAO60GbkmS1KC2F6DbanjCVuB84FXAQmBGRFyZmXPKdjsemFw8XgJ8r3hWAxnSv41D9xzJoXuOfMb69Zu2sOjJdSxYsZYFT6xj4Yq1PLZ6A8tWb2DBirXc+fATrFi7ke39jhYBg/u1MahfK4P7F8/92hjcv5WB/VoZ0NZK//YW+re1MqC9lQHF6/bWoH9bC+2txaOthX6tQWtLC22tQVtL0Fa8bm0JWqP03FI8b3u0BLRE0FL+OoIoXgfFcwt/fx0QFM/lr+HpoS1RfDaHukiS1JxqFqCBw4B5mTkfICIuBU4AygP0CcCPs9QNfltEjIiIcZm5pIZ1qZsMaG9lnzFD2GfMkO3us2nLVlas2ciTazexcl3p8eTajaxct4lV6zbx1IYtrN24mTUbt7B2w2bWbNzM8qc2sm7TFtZv2sL6TVvZsHkLGzZtZeOWrT346brftjwdTy9Hp+WyfXnGQtfHq+Bcz35P1xvqlfX9FUOSVInrP/6yhprcoJYBugNYULa8kGf3Lne1TwfwjAAdEacApxSLT0XE/d1basVGA8vrdG71DNu4b7Cd+wbbuW+wnZvf6D2+XLc23rOrlbUM0F11LnX+Y34l+5CZFwIXdkdRuyIiZnY1DkbNwzbuG2znvsF27hts5+bXiG3cUsNjLwQmlC2PBxY/h30kSZKkhlHLAD0DmBwRkyKiH3AScGWnfa4E3hklhwMrHf8sSZKkRlazIRyZuTkiTgOuoTSN3cWZOTsiTi22XwBcRWkKu3mUprF7T63q6SZ1H0aimrON+wbbuW+wnfsG27n5NVwb97obqUiSJEn1VMshHJIkSVLTMUBLkiRJVTBAVyAijouI+yNiXkScWe961D0iYkJE3BgR90XE7Ig4vVi/W0RcFxEPFs8jd3YsNbaIaI2Iv0TEr4tl27jJFDfiujwi5hb/Tx9hOzefiPhY8fN6VkT8NCIG2M69X0RcHBGPRcSssnXbbdeI+FSRye6PiNfUo2YD9E6U3ZL8eGB/4OSI2L++VambbAY+npn7AYcDHyra9kzghsycDNxQLKt3Ox24r2zZNm4+5wC/zcwpwEGU2tt2biIR0QF8BJiWmVMpTVBwErZzM/hP4O20ye4AAATJSURBVLhO67ps1+Lf6ZOAFxbv+W6R1XqUAXrnnr4leWZuBLbdkly9XGYuycw7i9erKf2D20GpfX9U7PYj4A31qVDdISLGA/8AXFS22jZuIhExDJgO/AAgMzdm5pPYzs2oDRgYEW3AIEr3jrCde7nMvBlY0Wn19tr1BODSzNyQmQ9RmsntsB4ptIwBeue2d7txNZGI2At4EXA78Lxt85EXz2PrV5m6wdnAJ4GtZets4+ayN7AM+GExVOeiiBiM7dxUMnMR8E3gEWAJpXtHXIvt3Ky2164NkcsM0DtX0e3G1XtFxBDgF8BHM3NVvetR94mIfwQey8w76l2LaqoNOAT4Xma+CFiDf8ZvOsUY2BOAScAewOCIeHt9q1IdNEQuM0DvnLcbb2IR0U4pPF+Smb8sVj8aEeOK7eOAx+pVn3bZS4HXR8TfKA2/ekVE/Be2cbNZCCzMzNuL5cspBWrbubkcCzyUmcsycxPwS+BIbOdmtb12bYhcZoDeuUpuSa5eKCKC0pjJ+zLz22WbrgTeVbx+F3BFT9em7pGZn8rM8Zm5F6X/d3+XmW/HNm4qmbkUWBARLyhWvRKYg+3cbB4BDo+IQcXP71dSunbFdm5O22vXK4GTIqJ/REwCJgN/7univBNhBSLitZTGUW67JflX61ySukFEHAX8AbiXv4+P/b+UxkFfBkyk9AP7xMzsfHGDepmIOAb4RGb+Y0SMwjZuKhFxMKULRfsB84H3UOoksp2bSER8EXgLpVmU/gK8HxiC7dyrRcRPgWOA0cCjwOeBX7Gddo2ITwPvpfTfwUcz8+oer9kALUmSJFXOIRySJElSFQzQkiRJUhUM0JIkSVIVDNCSJElSFQzQkiRJUhXa6l2AJKmkmF7vhmJxd2ALpVtUA6zNzCNrdN69gCMz8ye1OL4kNRunsZOkBhQRXwCeysxv9sC5jqGYI7vW55KkZuAQDknqBSLiqeL5mIj4fURcFhEPRMRZEfG2iPhzRNwbEfsU+42JiF9ExIzi8dJi/csi4q7i8ZeIGAqcBRxdrPtYRLRGxDeK990TER8sO/fNEfHfETEnIi6ICP8dkdTnOIRDknqfg4D9gBWU7rp3UWYeFhGnAx8GPgqcA3wnM2+JiInANcV7PgF8KDNvjYghwHrgTMp6oCPiFGBlZr44IvoDt0bEtcW5DwP2Bx4Gfgu8Ebi8Rz61JDUIA7Qk9T4zMnMJQET8FdgWbu8FXl68PhbYPyK2vWdY0dt8K/DtiLgE+GVmLizbZ5tXAwdGxJuK5eHAZGAj8OfMnF+c+6fAURigJfUxBmhJ6n02lL3eWra8lb//XG8BjsjMdZ3ee1ZE/AZ4LXBbRBzbxfED+HBmXvOMlaWx0p0vnPFCGkl9jmPXJKk5XQuctm0hIg4unvfJzHsz82vATGAKsBoYWvbea4D/FRHtxXv2jYjBxbbDImJSMfb5LcAttf8oktRYDNCS1Jw+AkwrLgKcA5xarP9oRMyKiLuBdcDVwD3A5oi4OyI+BlwEzAHujIhZwPf5e8/2nyhddDgLeAj47x77RJLUIJzGTpJUEae7k6QSe6AlSZKkKtgDLUmSJFXBHmhJkiSpCgZoSZIkqQoGaEmSJKkKBmhJkiSpCgZoSZIkqQr/H49U729eRbJEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### Initialize\n",
    "environment.reset()\n",
    "\n",
    "## Creation of the environment via Environment.create() creates\n",
    "## a wrapper class around the original Environment defined here.\n",
    "## That wrapper mainly keeps track of the number of timesteps.\n",
    "## In order to alter the attributes of your instance of the original\n",
    "## class, like to set the initial temp to a custom value, like here,\n",
    "## you need to access the `environment` member of this wrapped class.\n",
    "## That is why you see the way to set the current_temp like below.\n",
    "environment.environment.current_temp = np.array([0.5])\n",
    "states = environment.environment.current_temp\n",
    "\n",
    "internals = agent.initial_internals()\n",
    "terminal = False\n",
    "\n",
    "### Run an episode\n",
    "temp = [environment.environment.current_temp[0]]\n",
    "while not terminal:\n",
    "    actions, internals = agent.act(states=states, internals=internals, evaluation=True)\n",
    "    states, terminal, reward = environment.execute(actions=actions)\n",
    "    temp += [states[0]]\n",
    "    \n",
    "### Plot the run\n",
    "plt.figure(figsize=(12, 4))\n",
    "ax=plt.subplot()\n",
    "ax.set_ylim([0.0, 1.0])\n",
    "plt.plot(range(len(temp)), temp)\n",
    "plt.hlines(y=0.4, xmin=0, xmax=99, color='r')\n",
    "plt.hlines(y=0.6, xmin=0, xmax=99, color='r')\n",
    "plt.xlabel('Timestep')\n",
    "plt.ylabel('Temperature')\n",
    "plt.title('Temperature vs. Timestep')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the agent\n",
    "\n",
    "Here we train the agent against episodes of interacting with the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train for 200 episodes\n",
    "for _ in range(200):\n",
    "    states = environment.reset()\n",
    "    terminal = False\n",
    "    while not terminal:\n",
    "        actions = agent.act(states=states)\n",
    "        states, terminal, reward = environment.execute(actions=actions)\n",
    "        agent.observe(terminal=terminal, reward=reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check: Trained Agent Performance\n",
    "\n",
    "You can plainly see that this is toggling the heater on/off to keep the temperature within the target band!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAEWCAYAAABPDqCoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOy9eXxkV3Xv+1uahypJraFKPXerW1LbYBuCbQjEhgAJY8j0uAwZyUB495JwH+8m4QVIQhLuIxODY4YQSAgJCQRIDImNwQNmNrFNPPSkKrV67laVhm6pSrOq9v1jn3XOrlNn2NXdaqmt9f18+tMatnbtfc4e1l7TJqUUBEEQBEEQBEGwo269GyAIgiAIgiAI1xIiQAuCIAiCIAhCDYgALQiCIAiCIAg1IAK0IAiCIAiCINSACNCCIAiCIAiCUAMiQAuCIAiCIAhCDYgALQiCIKw5RPQgEb1uvdshCIJwJRABWhCEawIiKhr/ykS0YHz/c+vdvsuBiMaJ6EfWux2XChG91HgXc0SkfO8rpZR6sVLqc1e5Xdf0cxUEYePSsN4NEARBsEEpleCviegEgF9TSt2/fi2yg4galFKr1/pnROG8h4TTlgMADprvSxAE4emGaKAFQXhaQET1RPRuIhojokki+gwRdTm/O0BEq0T0q0R0loimiOhXiOiHieggEV0kovcbdb3FcTn4ayKaJaLDRHS78ftuIvq0o+E8TUR/QER1vr/9MBFdAPAO5/MfIqJpIpogor8noqRT/vMAUgC+5mhrf4uIXk5Eo77+udpUInofEf0TEX2OiAoAXh/V/4BnNUZELzW+byaiGSK6nojaieizTlsvEtH3iWjLFXg/DxPRz/ue0Z3O52aJ6GYierPzfnJE9Hrjb1uJ6IPOsx4nor8iombnd/1EdK/T1ikiejDsuTo/v83p00Ui+gERvcDXxj8mosecdn2RiDovt++CIDz9EAFaEISnC78N4McB/AiAHQBWAHzA+H09gBsBDAB4E4C/AvC/ALzQ+fmbiOi5RvnbATwBoAfA+wDcRUQdzu8+A2DGqetWAD8F4Bd8f/s4gF4Af+n87I8A9AO4AcAwgHcCgFLqtQDyAH5cKZVQSt1h2d+fBfD3ADoBfNGi/yafBfAG4/tXAziulDoM4NegrZPbnfa/FcCyZZtq4TYA34V+vndB9+E6AHsB/DqAjxJRi1P2A9B94mc3BOAdzu9+F8CI09atAP4QCH6uRLTH+ax3AugG8C7o92oeEH4RwM9B978J3vsTBEFwEQFaEISnC78B4B1KqXNKqUUA7wHwOiIio8wfKaWWlFJfdr7/tFJqSil1ClqYe7ZR9rRS6iNKqRWl1KcBnAHwMiLaDS0gv10pNa+UOg/gDgCvN/52TCn1N0qpklJqQSl1VCn1oFJqWSk1DuCD0IL75fANpdQ9SqmyUmrBsv/MZwD8DBE1Od+/EcA/OV+vAOgDsE8ptaqUekQpNXeZbQ3iqFLqnxzXk38BsAvAHzrP6MvQwuseImoA8CsA3qaUuqiUmoE+0PDzXgGwDcAu52+/GfGZvwTgX5VS9zvP7R4Ah6EPHszfOe+rCOAPUHnQEARBACA+0IIgPA1whMSdAO4hImX8qg5awwkAJaXUlPG7BQA53/em3+4Z38echBbUdgNoATBhyKZ1AEyXi9O+9m0D8CEAzweQdMqft+lbBO5nWPR/0vxDpdQhIjoN4BVE9HUArwDwP51ffxJaU/4FIkoA+DSAdyulSpfZXj/+Z7/kCMfmzxLQz7wRwCHjeRMA9vl+L7R2/+tEtALgI0qp9yOY3QDeQESvNX7W6HwGY767kwDaiKjT1zZBEDY5IkALgnDNo5RSRHQWwM8opR7z/56Iei+h2h2+73cBOActYBUBbFFKqaq/cprk+/7PAcwBeKZS6oLj3/snEeXnALTxN0TUCO1yEPgZcf0P4Z+htavdAB5RSp126loC8PsAfp+IBgB8FcAhaK31enAeWlje5zsAAQAcwfZtAN5GRDdBC9LfV0p9B9XP9TSATyilfjPi83YaX+8CMC/CsyAIfsSFQxCEpwsfA/A+ItoJAESUIqKfuIz6djrBbg1O8NsuAF9TSh0H8DCAPyOiJBHVEdEgRadLS0IL3bNEtAvA232/z0H7UzNHAHQT0Usc4fk9iF+va+3/P0P7Pv8aPPcNTkl3PemgyFlo4fVKa5+tUUqtAPhbAB8iol7S7CSiH3Pa+xoi2uto4WectnJ7/c/17wG81nmu9U5w4kuIqN8o88tENORo3/8QwFVNvScIwrWBCNCCIDxd+DMA9wN4kHRmiu8C+KHLqO+b0D7R09BBZz9taCLfAKALwFHn958DkI6o6/ehg/tmAPwbdMCcyXsBvNfJDPFWpdQktFb1M9CuJOPwuWEEUFP/lVInoIMkbwHweeNX2wF8CUABwEEA90D7KIOIPkVEH4xpx1rwP6G1/49CP8N7Aex3fncdgIeg2/tNAH+hlHrY+Z3/uY5BB1++B/p5noR+zuZe+A/Qh4uzAMoA/t+165YgCNcqFG6BFARB2JwQ0VsA/F9KqZfGFhaeNhDRwwDuVEr943q3RRCEjY1ooAVBEARBEAShBtZMgCaivyWiPBEdDPk9EdEdRDRKRE8S0eWYWgVBEARBEAThqrBmLhykb+0qQudZfWbA718J4DcBvBLAcwF8SCn1XH85QRAEQRAEQdhIrJkG2klmPx1R5CehhWvlBHx0EdHWtWqPIAiCIAiCIFwJ1jMP9HZUJqw/4/ys6nIBInozgDcDQHt7+3MOHDhwVRrIrJQUjo7PYntXK7rbm+L/QBAEQRAEQbjmeeyxxyaVUn3+n6+nAB10vWygP4lS6uMAPg4AN998s3r00UfXsl1VrJbKGHzXV/DWFw/i7T82dFU/WxAEQRAEQVgfiOhk0M/XMwvHGVTe+LQDOs/nhqOhvg497U2YKCyud1MEQRAEQRCEdWY9BegvA/hFJxvH8wDMKKWq3Dc2CqlkC/KzS+vdDEEQBEEQBGGdWTMXDiL6ZwAvAtBLRGcA/AGARgBQSn0M+narVwIYBTAP4E1r1ZYrQaqjGTnRQAuCIAiCIGx61kyAVkq9Ieb3CsD/WKvPv9Kkks04fG52vZshCIIgCIIgrDNyE6El6Y4WTBaXUCrL1eeCIAiCIAibGRGgLUklm1FWwNSc+EELgiAIgiBsZkSAtqQv2QIAEkgoCIIgCIKwyREB2pJURzMAIC+BhIIgCIIgCJsaEaAtSXeIBloQBEEQBEEQAdqavoTWQOdEgBYEQRAEQdjUiABtSVNDHba0NYoLhyAIgiAIwiZHBOgaSHe0IF8QDbQgCIIgCMJmRgToGuhLNosALQiCIAiCsMkRAboGUskW5GfFhUMQBEEQBGEzIwJ0DaQ6mjFRWEJZbiMUBEEQBEHYtIgAXQPpZDNWywoX5pfXuymCIAiCIAjCOiECdA2knFzQkspOEARBEARh8yICdA2kknIboSAIgiAIwmZHBOgacG8jlEwcgiAIgiAImxYRoGugz9FAT4gALQiCIAiCsGkRAboGWhrr0dHSgJykshMEQRAEQdi0iABdI6mOFuQliFAQBEEQBGHTIgJ0jaQ7miWIUBAEQRAEYRMjAnSNpJItEkQoCIIgCIKwiREBukZSyWbkZ5eglNxGKAiCIAiCsBkRAbpGUh0tWC6VMbOwst5NEQRBEARBENYBEaBrxLtMRdw4BEEQBEEQNiMiQNcIC9CSyk4QBEEQBGFzIgJ0jaT4NkJJZScIgiAIgrApEQG6RsSFQxAEQRAEYXMjAnSNtDc3INHcILmgBUEQBEEQNikiQF8CnMpOEARBEARB2HyIAH0JpOQ2QkEQBEEQhE2LCNCXgNxGKAiCIAiCsHkRAfoSSCWbkZtdlNsIBUEQBEEQNiEiQF8CqY5mLK6UUVhaXe+mCIIgCIIgCFcZEaAvgbTkghYEQRAEQdi0iAB9CfS5uaAlkFAQBEEQBGGzIQL0JZBKigZaEARBEARhs7KmAjQRvZyIRoholIjeEfD7TiL6dyJ6gogOEdGb1rI9V4p0h2igBUEQBEEQNitrJkATUT2ADwN4BYDrAbyBiK73FfsfAA4rpW4C8CIAf0lETWvVpitForkBrY31ooEWBEEQBEHYhKylBvpWAKNKqTGl1DKAzwL4SV8ZBSBJRAQgAWAawIZPbUFEzmUqIkALgiAIgiBsNtZSgN4O4LTx/RnnZyZ3ArgOwDkATwF4m1Kq7K+IiN5MRI8S0aMTExNr1d6a4FzQgiAIgiAIwuZiLQVoCviZ/+aRlwF4HMA2AM8CcCcRdVT9kVIfV0rdrJS6ua+v78q39BJIdbRgQjTQgiAIgiAIm461FKDPANhpfL8DWtNs8iYA/6o0owCOAziwhm26YqSS4sIhCIIgCIKwGVlLAfoRAINEtNcJDHw9gC/7ypwC8BIAIKI0gGEAY2vYpitGKtmC4tIq5uQ2QkEQBEEQhE1Fw1pVrJRaJaK3AvgqgHoAf6uUOkREb3F+/zEAfwzgU0T0FLTLx+8qpSbXqk1XEi+V3RL2Nq/ZYxQEQRAEQRA2GGsq+Sml7gFwj+9nHzO+Pgfgx9eyDWuFd5nKIvb2tq9zawRBEARBEISrhdxEeImkDA20IAiCIAiCsHkQAfoSSSW1AC2p7ARBEARBEDYXIkBfIp2tjWhqqJNUdoIgCIIgCJsMEaAvESKSVHaCIAiCIAibEBGgLwO5jVAQBEEQBGHzIQL0ZZDuaBENtCAIgiAIwiZDBOjLIJVsRl400IIgCIIgCJsKEaAvg1RHC2YXV7G4UlrvpgiCIAiCIAhXCRGgL4M+J5VdflbcOARBEARBEDYLIkBfBukO5zbCgrhxCIIgCIIgbBZEgL4M+DIVCSQUBEEQBEHYPIgAfRnIbYSCIAiCIAibDxGgL4Pu9ia0NNbh7IWF9W6KIAiCIAiCcJUQAfoyICLs6m7Dyen59W6KIAiCIAiCcJUQAfoy2dXdjlNTIkALgiAIgiBsFkSAvkz29LTh5PQclFLr3RRBEARBEAThKiAC9GWyu6cNiytlycQhCIIgCIKwSRAB+jLZ1dMOADgpbhyCIAiCIAibAhGgL5Pd3W0AgJNTc+vcEkEQBEEQBOFqIAL0ZbJ9Syvq60g00IIgCIIgCJsEEaAvk8b6OmzvapVUdoIgCIIgCJsEEaCvALt72nBKXDgEQRAEQRA2BSJAXwF2dbfhhLhwCIIgCIIgbApEgL4C7Olpx8zCCmbmV9a7KYIgCIIgCMIaIwL0FWBXj5OJY1rcOARBEARBEJ7uiAB9BdjNArSFG8e9B8fx+OmLa90kQRAEQRAEYY0QAfoKsMsyF7RSCr/zhSfwofszV6NZgiAIgiAIwhogAvQVoK2pAalkc6wG+tT0PGYXVzE2Ka4egiAIgiAI1ypWAjQR7SCiH3W+biai9rVt1rXH7p622FzQT56ZAQCcnp7H0mrpajRLEARBEARBuMLECtBE9CsAvgzgE86PdgP40lo26lpkV3d7rAvHU2e1AF1WWogWBEEQBEEQrj1sNNC/BeB5AGYBQCmVAZBay0Zdi+zpaUNudgmLK+Ga5SfPXERbUz0A4NiEuHEIgiAIgiBci9gI0ItKqWX+hojqAdDaNenahFPZnQrRLJfLCgfPzuJlz+gHAIyJAC0IgiAIgnBNYiNAf4eIfgdAi+MH/TkA/7G2zbr22N2j3cLDAgmPT82huLSKH97Xg75kM8Ymilb1Ts8t49zFhSvWTkEQBEEQBOHysBGgfwdAAcBRAG8D8ACAd65lo65FdseksnvKCSC8cUcnBnrbrTNxvPPfnsKv/v2jV6aRgiAIgiAIwmUTKUA77hp/q5T6qFLqp5VSP+V8XbapnIheTkQjRDRKRO8IKfMiInqciA4R0TcuoQ8bgq62RnS0NIRqoJ88M4OWxjrs70tgoK8dxy0F6MdPX0Q2V8BKyeqRC4IgCIIgCGtMpACtlCoB2EpEjbVW7AjfHwbwCgDXA3gDEV3vK9MF4CMAXqOUegaA19b6ORsFIsLunvbQVHZPnb2IZ2zrREN9HQZ6E5ieW8bF+eXAssyFuWWcn1nEalnFZvgQBEEQBEEQrg4NFmXGAHyLiL4EwJXilFJ3xPzdrQBGlVJjAEBEnwXwkwAOG2XeCOBflVKnnDrzNbT96vOiF0X+etfgT+Bgexp40e9U/LwEwsFbfguvyz8FfO73MNA1ABz4WRz7b7+M5xTPhdZ3pGMXcP3rAACjv/F27L+QjW3iqeZOfL9jB147cSi+P4IgCIIgCBuFhx5a7xZYY+MDPQHgPgBtAPqMf3FsB3Da+P6M8zOTIQBbiOghInqMiH4xqCIiejMRPUpEj05MTFh89Pqwe/EizjZ1YNWXpORYazcW6ptw49w4AGBgcRoAMNa6JbK+w23eYx5t7bFqwye33ozf3vdKzNQ319J0QRAEQRAEwZJYDbRS6t2XWHdQqjsV8PnPAfASAK0AvkdEDzu5ps02fBzAxwHg5ptv9tdx9Yg5Ge155DRWv/gkzn3xbjetHQA8+dgZ4PNP4MaP/CmQSmJnqYyGd9+LsV/9TeDlB0LrO/wvjyOVnURDHWH0Na8DXv++2CYe/evvAcenMfq3n8VzdkcL6IDO8vE33xrD//PSITQ1yM3ugiAIgiAIcdjcRHgfEX3N/8+i7jMAdhrf7wDg91c4A+BepdScUmoSwDcB3GTb+I0GC80npyv9lZ86cxHtTfXY25sAADTU12FXT1tsKrsj5wu4bmsH9qUSGLVIe6eUwtHxAgBgNF+wavPdT53HRx86hh+cumBVfnm1jP//K0eQLyxalQeAuaVVlMrrd+4RBEEQBEG4ktioHN8F4N3Ov/dCp7N7wuLvHgEwSER7iagJwOuhrwQ3+RKA24iogYjaADwXwBHbxm80djsC9AlfJo4nz87gGds7UV/nKeUHehORmTiWV8sYzWsBen8qgWP5OZRjhNDc7BJmFlYAANmcXZ7pjCNwZ3N2Avfjpy/ir78xhnuePG9VvlRWeOGfP4RPfnvMqjwAvPfuw/jrbxyzLn/vwfO488F4/3AmkyvgT+89CqXshPqZhRW89+7DWFgOv2XSpFRW+MB9GZyfsc/f/Znvn8STZy5al7//cA4PHs1Zl3/i9EV87pFT1uXPzyzgw18fjR1zzOJKCe//2gjmllatyiul8LFvHKvpSvsvPnYGj520O+gBwDczE7j34Lh1+SPnZ/EPD5+0Lj9ZXMKH7s9aHw6XV8t4/30ZzMyvWH/GJ799HMcsc8YDwJefOIeHx6asyz88NoUvPxEeh+Hn2EQRn/z2cevyM/MreP99GSyv2mURKpUVPnR/FpPFJevP+IeHT+LI+Vnr8vceHMc3M/augI+dvIAvPnbGuvzp6Xl87BvHrNeXuaVVvP9rI5G32JqUywof/vpoTevL5x45VdP68vWjeTxwxH59OXh2Bv/8n/brS252EXc+mK15fSnWsL58/JvHcCokK1YQ//ZfZ/DoiWnr8t/OTuIrT9ntgwAwMl7Ap793wrr89NwyPnh/BquWGbhWSmV84L5MbHICk7/7znFrZRsA3P3keXz32KR1+f88Po0vPX7WuvzxyTl84ltj1nNnvYkVoJVS3zf+fUMp9VvQAYJxf7cK4K0AvgotFP+LUuoQEb2FiN7ilDkC4F4ATwL4TwCfUEodvIz+rCvpZAuaGupwysiYsVIq4/C5Wdy4vbOi7L6+dpyYmg/dfEfzRayUFK7fpgXohZUSzs9Ga32PjutNpKGOkM3bbbojjgCdsRS4RxxBO2NZ/8mpOUwWl/DoCTvBRymFzz92Bnc9br+p/8PDJ3FnDcLe5x89jY8+dAynp+02oAeO5PA33zpuvXAcHZ/Fhx7I4t/+y27hWFot4fe/dKgmweR99x7Fn907Yl3+498aw7vuOmidDvELj57Bn391xMryAWhh9Y4HR/HgUbs44NPTC3jfV47ic4+cji8MLTS8666D+OhD9gerv7wvg/feczi+oMOnvnMC777roPUh4EuPn8MH7s9YCyaPnJjGHQ9k8ZWDdpvuZHEJf/wfh/GPNQj17/nyIdzxgP1h8o4HsnjPl+0Djv/x4ZP44/84bC3gfuXgedzxQBaPWAomT565iA/cn8GXLOf/3NIq3n3XQXzqOyesygPAe+85jL+8LxNf0OGjDx3Du+46aL2+fO6R03jfV45ary8PHs3jjgdHrYX60Yki/vyrI/jCo3ZC/UqpjHfddRAf/6a9EuNP7z2K933lqHX5v3HWl6VVu0PAF39wBn/xtYxrMY3ju8cmcceDo7j/sJ1Qf/biAv73PUfxT5ZCvVIK777rED789VGr8gDwgfsz+JO77fV9n/ruCfz+lw5hdtHuAP3vT5zDB+/P4vHTduvLYycv4EMPZHG3pVA/PbeM9/z7YXz6ezWsL/9+CB+83359+asHs/iDLx+yFog/8/BJ/MndR5Av2B+g1xMbF44O418XEb0EwFabypVS9yilhpRS+5RS73V+9jGl1MeMMn+ulLpeKfVMpdQHL7knG4C6OsLu7raKXNDZXBFLq2XcsKNSgB7oa8fyahlnLwQvsqxRuX5rEvv7tOvHaIzQyovRbYO9sWUBvWiwQJy1PIXWqrHOsMBtWX6isISL8ys4li9an7wzuSIWV8o4fcFO28CHBds2eeUttfrcZ8vNYWxiDqWysq5/ebWME5NzGJuYs35GOpe4fTrEkRrfG5ezHRe11n/mwgIWVkrW47RcVsjmCjg9vYD5ZTuBmNtkM3cAcy6szWG11voni0uYmlu2rl+3pYCpuWVMWQrEtc7ntR5HrCjIWI6LuaVVnJ5ewGiuYL2pZ3IFLKyUcNbyRthL7nOtSg/L8icm57BSUtbjaLVUxtjEHI5PzllbDkbGCyiVlfX9Bu7YtnxvI+O1rtm1jaNzM4soLq1azx2lFDLjBZy9uGCtFffaVKOyqsZ9x9r6XOM4vTi/jHxhCdka587F+RVMWK4vtc6d9cbGheMQgIPO//8FfQvhr69lo65ldvdUCtBPndWnxxt3dFWUG3CE4mOTwYP98PlZtDTWYW9vAvtTdgL0yHgBWztbcPOebquJzS4fTfV1lzSpbSYRT/6T0/NWJkquf7lUDs2pbXJhbhkTzmnVdqHhRdV20611oblUgdv20HB8cg6rZWX9jJZX9YZYS5uyNffh0vpsKzRw+VPT81auNGcvLmDeKWd7mMzW+p7ztZVnYcH6sHqJgthkcQnTc/Fm3KniEiaLy87f1vaebdeLSx9HtQpKdusRj4W5ZTuBeH551T2YW7/nNV5fsjUKh/xMxyaLVhaoE1PzWC6VsVpWOGFx4GaB2/ws2zbV/Exrrb/Gd2ArEI/PLqLglLN5D0qp2g+Hlzj/ax9HtT3TC/Mr7roRxcz8CnKzSzV9Rq3rxXpjI0APKKV2KaV2KqX2KqVeDOA7a92wa5Vd3e04NT3vLuZPnplBsqXBveqb2dvbDgDuwuPnyPlZDKeTqK8j9CSasaWt0UoDPdyftBe4nQn0wuE+Ky0ULwJNDXWYWVhxBVebz1DKTpAxJ47NQmMuFjYLx+ziCs7NaFcYWw1xzYIMB3JOFK38Y7leW4F4xOyzRR9OTGmBG/C0V1GslMoYcw52a/aMnHInpuZqOljZjyOvHTZ9PntxAXOOwG3TBy1wFyvaFge3w6Y9ul5df76wZOXXaL4rmz6Yc82mvHlYte7zJWugLQ/oTp+LS6vuvLZpj22bRvNFcDNs+rywXHLnsP3cccaR9bhwDtwTdgIxl18pKZyw0BDXOndY4Abs+lwqK9c1jDXLcdQ8jpx2nJ5esHLJMttts++M1DjXxmcXUVjU7bAZR0opw2Jl2+faDiXcjqm5ZSuXrFrnjnl4sRlHMwsrGJ+tbW9eb2wE6O8H/Ow/r3RDni7s7mnDwkrJ3WieOjuDG7Z3oq6uMqtfT3sTOloacDxAA62UwuHzs7h+W4f7Mx1IGL7YrJR00OFwfxKDjgAdtxDwIH31jdojJ04byO4VLxrS+altTomZ8QL2OMGVNpq3zHgByZYGENktrmzGbG2st5rULPQ0N9RZtX9uaRVnLiygqb4Oo3lLgThfQFN9HZZXy1YuE5lcEY315LTPpg8F1NcRiCzfgVNnYz1ZvQM2+TbWk5UGhzVQjfVkLRBzn20F4myu4D6jWoRD23gAHheN9WT1TNnk21hPVtoVFrgb6wn5wpJVIGFln+3mQk3jKO+NC7tnaowji/IX57XAzfXbWax0nwtLqzhvIRCbfbab/wU01NXwTI1xYfOej01ogdt2HC2ulHBySs8dW5csHke2Llk1j6NcAURAHdmvR4D9ODo5pV1DbNejUllh1HnPpy/YWaAyea/Ptoobb+5c+fWi1nGULyxhdnHVun6llNvnyeKylQXK7LPt3KltDa5t36kYRzUENq4noQI0EaWI6CYArUR0AxHd6Pz7EehLVYQAzEwcS6slHDk/W+X/DOirvwf6EoEa6PHZRVycX8F1Wz0Bel9fdCq7447Qc11/B3Z1t7kCXxRHxwtIJZtx695uAPGLJZ9AX+UI3HGTaHm1jOOTc3jZM/rtF5p8Ac/YpvtgM4ky4wUkmxtw697umoTJHx1OWWmIWfh64XAfllbLOBWjIZ5f1j6Wt9dyyMgVcNugLm9zaBhxDiW7utvsFrLxAuoIeP6+XitNAL/n2wf7cHIq3vWGNVC3D/ahrBCbNWK1VMaxiSJud/psdbDKFfHcvT1oqq+zGhfZnHZnGkwna1rsbxvsq8nycftgH8ZnF93sN2GwyZf7HNcHtva45S03uJt2dCHZ0mA97jpaGnDjji6rTZ0Pq7cP9lm5cHEbbh/sQ2Fx1TXnhsEm31r7zHPH7r0VMZROor+jxbr+poY6Z32pbVzYrC+j+SLKSpe3sUAtrpRwYmrOeEZ27/n5+3pRR3baz2yuiN3dbdjT224tHBIBL9jfa3VYNceFjUvW6el5LK3q9cXmwM0Cd03jKF/ALXu6HcWK3XvuSzbjQH+H9TgC9PpS6ziyccmaKGrllm2f2WXNXYMtx9Ezt3eiq63Ralxkc0W0N9Xj2bu21HTIuH2wz9oCtd5EaSCN1ukAACAASURBVKBfBeBO6PzNHwHwYeff70GntBMC2N2jXTNOTs0hM64zady4vSuw7EBfe6AAfficDiA0Bej9qQSm58JPlhx0ONyfREN9HQb62mMXs0xOa6z7O1qQbI7fdPn3z9/Xiy1tjbGCD/vqXr+tA3t722PNMqylG0onMZhKWgsyg+kEhvuTVj7EmVwBrY31ePGBlJWG2K+lj1uYeCF61Y39VuUXlks4NT2Pm3Z0WR8asnnvGdlqY/f0tOOG7Z3uwS6ufB0Br7hhK0plFepm5JV3ntFNjiUjZhydnJ7H8moZP/6MNBrrKfbQwCbf67YmMdAXP44ALSgMppMYSiesy6c7mnHzni04N7MYGynPdfJhMvbwOV75jOIOMrlZrYG6fajPmZvxc2dkvICh/iSG0kkrQSkzrscRl48ViJ3D6u1DfZhZWImNlB/xjYu4NvHYtz2gzyys4PzMIm7Z041UstnOYpUrYCidwGA6YTV3RnIF7OtL4Lr+DisL1EhOW59+7Pq01frCayg/o7ixemxCC9yvuGGrY6WLLr+0WsKJqXncuKMTu3va7dwTeO5Yry8F7NzShpt2dFlZoLjOV96w1UogrnUcnZ6ex+JKGS+5Lm0lEOuA4yIO9OuMV1Zzp9ZxNF5Ab6IJzxvotnLJGhmvbS6w+4Y7jmLKTxSXcGF+BS/Y34vO1kbLPhcxXMO4GBnX42g4nURm3GJ9yRXQ1lSPFx1IWbtkrTehArRS6u+UUrcB+FWl1G3Gv1cqpT5/Fdt4TbG9qxV1pIOdnjo7AwC4YXu1BhrQWuXx2cUqHy0Whg/0J72yMX7NI+PaNLnPCU7cn0pECrg660MBw+kkiMhqIciMF9DT3oS+ZLOj2YsX0AFoYS+djBUO2Sw+5Ag+YxPRUeCspWO3FRsNDi98w86ztelDc0MdXnwgpb+P2bC4zzfu6MLO7tbYZ8o+lkPphJWwxybfoXQSw/0Jq0h53eckhvqTVpHy2VwBux2BG4jXELPJ9yXXpdFQF2/G5U38uq0dGOhNxG7qbPIddIS9uHfGGqihVAJD6STOzSyiECMQ88FtKJV0v48ikyuiL9mMW/Z0u9/H1Q8ALxxKob2pPt69ypg7+y3mJpt8dZ8TsZHybPLlQ4ZNTAMfVgfTiYo2hpHNFZBobjC0XHZ9vnVvN3oTzbHPlPPX6rmTjB2ns4ta4OZxZCMQ63Gh67exQGVzRQz0tbvKDxulRGM94SXXpa1csngc3bijE7u722L7zBl+BtN6jYx7Z0urJZyYnHPXI1uBmJ+RjUCcyRWws7sVN+3sdL+PgsfNiw+knYB3u3F0YGsS+/oSsc+UM/y44yimfLmskM0XMZjS4yg3G++SlXHKD6bt9p1sroie9iY8b6DH+d6uzz+yv8+xQNkpenivjaufteCD6aQrK8QJxNl8wR1HhaVV1785qg+DqQSG3We08d04bPJA/wsRvYyI3k5Ev8f/rkbjrkWaGuqwfUsrTk7N46mzF9HZ2oid3a2BZTmQ0C/QHDlfwK7uNiRbGt2fxaWyGxnXmhK+jnswlcSZC+EpvE45ZrEhR5AcTCWtNAFDzuAesphEmZz21R3oa8dQKhmbUoyFx2FHixYXBT5ZXMaF+RUMppKuQBy/0BQxmPYCLWMPDfkiBtMJJFsasWNLa2zqKA6y3N3dhmGLxZg/f9A5ZMQJxGzyZc3haoxAzCZfXsiAeK3ViLOQ7e1tR0MdxZbP5ArY3d2GjpZGrSGO1cZqk+/+VAJD/fHaUq5vOK3fc1ykvDm2hyw2LL0hFpxDid3izYfP7V2taG+K978fcUy+3e1NGLTQEHsCtN5Q4jZcfkfc5wsxqaM4nmE47W1YUW0yD6tu+bhxNK4F7p5EM3oTzfHjaLyA9qZ6bO9qxXB/vLDHGmeeC9lcMTJXc9YcR+mkTn0ZIRAXFldw9uKCe/jkPkW3qeAoDCzXl/ECBnoT6GhptHLJGslpP9Q9Pe16HFke6HnuxFmg2GrIfY5zyWI3PT7Qm58Z1abhdBK7e9q1S1Zsn4vYsaUVna2N2GehIXbX1JRWlNg8U8CbO3EuWZzhx5wLUcohTqlZMXcs2jSUTmJrp7YO2/S5u70JvYkmR+Nru17oQ8NIjIbY3ZudcVRYjBaIOcMPz03zM6P6wAK9+ZkbGZs80B8B8EsA3g6gFcDPA9i/xu26ptnd3Y6TU3N48swMbtzRCSIKLDfQ52Ti8AlAh8/P4nrDfQPQmu3WxvpQIZczcDCD6QSUCs/ywYOZtdyD6URkJg5zEQD0phU3idhXt7mh3l1co4R0V2hImYJP+CRyNwcj80iUGZezCAynk2hvbtACscUGx1pJLRDHC+j7+hJoqK/DYDoZmzqKAw739GiBO+7QkDW0boOp+GfEJt/BdBJ7e9tRXxcdxKI13PMYSifR1FCnXW8stGisWbGyTOS1ybetqQFDqQTOXIiOlOf69qcSVgGypvaWF+Oo8qcvaJPvUDrhzrOoZ1p2NNyD6QTq6gj7Lfyss46WTrcrYXWw6k00oSehrT3TMZHylX2O16JnDA2UjVbMPKz2JJrR094U24dsvujOnaF0wuLwqccRETkuXNECMZt8t3e1YiitL5s6E5JX3+yfrYDLLnBDabvA7LmlVUfgTqCtqcHKAqWtALpuG5esbK6Avb3taGqoczTE0QJxJqctk3t7tcAd55KVyVUeSvRnRsfesMC9u6c9NtZlpaQF7sF0Eo2Oq6Hd3PHGkY11aHtXK5ItjRhMJ3A+xiXLFLht1gvzcGszjljgHkwnXIE4qn6lHAtaOmFYh+PXi8EUl9fW3iiBOJsvoKutEX2JZgylk5hdXI10yarocyp+vah1HOn1TQvcXW1NSCXjLVAbAZssHD+ilHojgCml1Luhr9vesbbNurbZ1dOGsYk5jIwXQt03AGBPTzuIgDHjhD+3tIoTU3MV/s+AvqRloK89MJBw1tGUHNjqCdDuQhByMh4ZL7haQF0+elJwmi9vIYufRNl80RW4bTZp9kPtbNOazDqKPoV62lu9YcX5EJvlAcRqiDmtjikcxqWOyuYKGDYEpbjUUZnxAgb62h2BO15DPDKuTb57etsx0KcF4kghwOnfcH8SzQ312NvbHqnNYJMva9yG+qNN42zyZc3KcDqJU9PzkZaGig3R+Zy4g9XO7la0NzcYloaIcWdsiDu3tKGlMTrjirnY19VR7CbNJl+vz9EbHJt8zbkTlzqKg910/fEHJTb59jobYlx5d0PsT6I30YTu9qbITd3V3hoH6Ki5xibfIaP8aK4QrSHOF7xn2p+Mvbwk67ig1NWR+zlxfW5trMeOLa3unI6KEzE11u6BO6q8IXDz30WNI77UxetzvEuWOS6G0vEuWZlc0RW47caRZzXc4wrEdofVxvo67OuLdgfgDD/DRh/iBO6xibmKPp+9uBDpksUuJYA3d+LWCxa4bfY1/t2gYYGKrD/vjSMbd0nXlbHfmwtRLlkcO8RzczidiL28hMcREdmtF/miFriTzVaHDLfP/Ulscdw+bdcj/jvbfPnriY0AzSrGRSLqd77fs2Ytehqwu7sNhaVVrJYVbgzIwMG0NGrtiakRODpegFKoSGHHhKWyy/i0yYAOZmyI0DZmctpNpK2pAYC36IcNWk/bWylwh00idh3g0+puJzNInBDA9bY01mNPT7T2M5PzTtG6TdGLN29+plAfpSHO+vo83J+ITB3FOaYHjcUeiDbXZYyFb19fIjZ1VDanTb6N9XVoaazH7p5osy9roPY4wa1xz8jUcAPaGhAVKc8aqEHj0ACEC8R8qYunjbV5Rp4VgAXiqPJs8m1vbkBdHcVq9kw3Gv4/uv7K8kPpZGSkPGugqg+fwZ/BEfKm1g2IPkzqwC9drjfRhC1tjbF97nYEbq3xjTaNe32u1KKHbercVvM9R11ewiZf/ziKO0wOpVh7m6hoZ2CbnGdUV0dINDdge1drbP0scHMfIg/0bp/t1heeI+Y4inLJml9exanp+apxFNWHjDGO2CUralyYVkPPAhVdfx151lT7ueO95yiXrJNTc1gulavWi7CDz2qAwM3tDG9T0W3P9q5WtMW4ZGVyOsNPR0ujq/GNG0e6z3aHhqpxlIp2yTo/ozP8+PedMDcOvkVxqIa5xpZYInJcsppi+qwz/KSS3t5sdxDzrDFxFqiNgI0AfQ8RdQH4CwCPAzgB4Atr2ahrHc7EAQA37AjOwMEM9CXcSysAL4DwOkObzOzv04tNVdCh6zvsCd2N9XrxC1toRnKetgcA0h3NjmkpvDzgLQLsbxU2KTg4joXDhhhzXcnwQ2XitFzmKZrbFhV4yFkE+jtanPqjNcSupiGVrPg/zE3EDMwAPIE4bLEsuibfykNDpBCQL7indCBey5XJaQ03+8YPpZORt0KywD3Q6y2uUYFBGV+fB2M2db7Uhcvv6m5Dc0N4YBAL3FxvXR1hf0wwlCl86jbFL97bu1qRaObDZAIThSVcCBGI/ZaMwZhN2r85xJk02drD9fclm9HZ2hiq/fQL3K4ZN2ZcsNDJbYoUiHPFisPqYDoZGSlvaibNvocf0CvH0X42E4eUZ5cWLp9sacS2zpYY03vRncPcpmgXjgL2pxJuDv/BdCJSIOaA453OpVlx68tIgNDA9QQx6mq4dXm2QIWNI87ww+OoqaEOe3rbI93cTEuJ7nP8ONrT046WxnrdthiXLM7ww8HuriUgdO5Ujos47Sen1OR641yyOKXmkLG+xAVb6oNY5TiK0pZm2bLaqmOa4lyyTFdGXX/0euGVt1uPOKUm1xvnksXxDzyOAMfdKNJ649ubU0lk8+ECcSZXvTfHuWRtBCIFaCKqA/AVpdRFJ/PGXgA3KKUkiDACzgXd096EbZ0tkWUHettxfGLO3bgOn59FR4vWjvhhdwu/D9vI+CySLQ1VnzWYTgQKPkurJRyfnKvwmY4zLWXGvVO3W38qfHH1b6D8ddgk5dRDQ2lzU0/gxGRwFDhParN8nA8xLwI8qePMdZmcF9QE6OevBeIYLb1PIA5b7E1XA2Ywwn2ATb5DFeWTkZHypn8yoPscJRCPjBexp9cQuDl4KmJcsMkXcCwNDXXhBzefdqXeEYhHQvrMAjdbAfhvw97Bim9DBPT7iEodNTJe8I27eIF4mzEX4kzj/sNnuqMZHS3hgUH+cUREbiqoIPzuVfy3Yamj/CZfQL/nOIHY3BDdYMuQNo3kiuhsbXQ1UN7BKnq94Ho7WxuxtbMltH6/yZe/DhtHbvyDOY76k5GXl3BAIDOcTsYKxIPpBOrrKteXsPfMOaZZ4RLnkuWfO80N9djT0xZav6vE8PUhTNjzAo4ry0e5ZJkuJYD3PsLmv75YyxO4Y+eOz9XQtUDFjSNDII46KHFKzSrFTYSiZzRfdN30uPxkMTx+yAy+r+hz6NzxXBkBHegHhCsl/Hstu2TZjiP+OmwccUpNc71gt5IggVgppfvsKz8fYYHKjBcx1O+tL3H7zkYhUoBWSpUBfMj4fkEpNb3mrbrG2eVoIG6ICCBkBvraMbdcch34j5yfxXVbOwL/jlPZ+aOiR8YLOGAMPmZ/KomTAcLVsbzj52pMIMARcEM11sWA8glnka6eRJwPlW8hBBCZQSFI4B5M6yjwsMtmCourFZtDVECHmUWAYYE4SpDZn066GijtMtEe6eZimny5TbGCkk+jHCYQ+02+gH4HYZHyrsnXp3UzP9uP6YcKeK434Roiz+QLaEuDTh0VLjSYJl/dpvDgTC/Ap3Kxz80uBUbKn5zSPpbBAnH1M/KbfCvKR2jdzXeQ7miOTB2VzRUrDp/sdxindRv0je1MLlgg9ls+9NfhqaP8Jl/A015FzZ2KZxqjLeWgSV6TOlsb0d8RriHO5CpNvrrP4Qf0rLteVL7nYyGXl/jddLgPy6UyTkxVZ+KYmdd5rm3HkW5TsWKuxVmgMrki9vd5AnecS1Y2X6wQuAFPkAmuv7rPg+lEqEuWl1Kzer0IOnB7KTVrOHzmKzWZOx0LVOgzzesMPyxws0tW1BpsCty6z7WPo7Db/DjDj18poT+7+jPclJoBzzRqvTDL9yWa0RVx7wKn1NzS3gQArktWVP1mu7lNYXt50Bo8mE6ECsQTBb02D6UqlWFhfeaUmuY7GIxYjzYSNi4c9xHRT655S55GtDc34BXP7MdrbtoWW5ZN5bzwj4wXqgIImT09WkNhLmZKqaoMHMxgKhEogI7kqvNMA95lLX7TEpu5/J8RZcY1g+PM9gDB5rfADc71dwsqXy1kuBtWwEndzCLAtDTWY1d3W6QgY2oauA9hmoBszsvMYPYh7Da/TK6IlsY67NziHTL40BAkELMg7he4dV3VbRp1fb69PnDqqCCh3m/yBRyBOGoxzlePi+F0eD7rTE5ruHlDBPQzCouUZ5OvuSF6gUHh4yJIKxbUh5PT846PpVc+KlI+aC64GuIIS0bV4bNflw/bsPo7WlyTL6DfeVikvN9FRH8dvqn7tXRm+aA+52aXqg6rnW2NSHcER8p7AndAn0OEANaIm0qA4XQi9Da/TK6IZItn8uU+hF1e4sY/mJrAiNSX3M4h34GbQixQHHBslo+zQPkP9Ny+KFO9KXADWqgJc8kyM/yY9YdZoLzAL7txxBl+zD5HuWRxhh/zHdTXxVg+AxU34RribK6IXd1taG3y1pfhdDLUJYtvUTTXl6j1ImjuuOMoYGzzLYpm+b6kFoiDDujlAFdGPnBHHQKGfc9IH6zC15fehE6pafY5dC8PWF+GLfZmc1xEJRHgWxTNPidbGrG9q9Xq4p/1xEaAfiuAfyOiBSKaJqILRCRa6Bg++vPPwc/8UHyyEjeV3cQcTk7NYX65FBhACMDNL2wufvqSiFUc6K/+GxaE/IPczORgEuZrFWTmMssHLzTVglVcedMPFUBkHmJ/oAUQHXgYpO3lvw9qj9/HkonKparzJ1fXH5Y6KuOU9wvcQLC/W9bxsWQLBwDs6Q1PHcXPzTxkcOqooPqDNFC6TcFZJvyBosxgOvzyEjMg0Kyf+1dVflxf6mIK3G62koDyI8615eaGuK2zBYmQ2/yCxhG7MwWNO54LptsN9zlIQ+xpoCrLD6USobf5+f0NAdP/PqDPuQJSyWZ0tRkbYoSZOGhD5Ej5INO43wXF/IygZ8om36pxlNLuSX6BmE2+/voHIwRiNoubAneUlovjH7Yabm77+rRAHDaOuI9MS2M9dofkag7SZOo+BFugvEtdqsdRmAUq43M14vaFCcSBSoyINZj3BVPDvbtHu3NFCZPmM4pyyTIvdanoQ0iQL7saBq1HYZeXBK3BUZbJkZyXUtOsP6w8zydzfUklHZeskLlptgFwBOJUsEuWmVLThC/Z8q8v5bJyLGLV46iwtIrzIQJx0Dgy++cvzyk1zfqBYJeskYBx0dHiuGQFPtNqpYf+jHDXvo2CjQDdC6ARQAJAn/N931o2ajPR39GClsY6jE3M4ch5Pbj8OaBN9qUSFansRsaDtcmAFkDrqHpxzeT0pSuN9ZWvP0zjayZRrywfLPiYFxCYRJnrgiZ1VB7ioFM04Ji6A0/F1QuZ7kOwQBykEefvgwRi9rEMW5iCT+rVgpJ7aAg5ZOxPVWqgGuvrQm/zc02+hsDNbbLdEPn7oNRR4QJ3sLbBvNTFpjzgmHx9wmpU6qhsXmeXMQVuovDAwyANFLcpSCDOhj6j4NRRQSbfyj5XtinI5Mv1B5XXbaouv8XJsBHW5z6fwM2fETROw/rMpnS/H2TUXFtarb68JMjka36ef1x4QZOV5b3LkYLXi/2GSwkAtDaxBSr4sJpoDoorCZs7lQHHZh+CLFDuMw04TAZZoAq+DD9M1OUlQdrbPT3h2ZDMDD9MvXO7bVj9ZoYfs8/B61HIOEoH3+Z3fJIF7pBx4Rur5i2KweWD37O/fH+HtkAF9jnvZfhhPJes4Pq5j5V9DnbJCrKsch+CXLLOXuRbFKsPq7q+yj74U2p65aOVW/5x3dmqLVCB7znnZfip7HP0vhP0nsNcsjYKNjcRlgC8FsDvOl9vBfCstW7YZqGujrC3V0d3Hz4/457gw9ifSji5NHXgCwvdQwECtA4yqdY2joS4fIT5co4E+JUBQFcb53esrN+fD5Xhvvnrd3N9BrRJ+2UHTzrT1MgMp5OBgYeZXBFbjCwCbv39wQJxkKlOtyd4YQoKagLCU0fNzK8gN1ut4eZDQ5ibi788EK7l4oNSQ9VBKThSPpOrNvnq8nwIqD6IAah6D2EuFkEmX8BLHeXX4CyuODmmfeWjUkf5A7/MNoUJVn6TL6D7fGF+BZPFSrOvl5IquM/+1FEjYYfPkJvtOJjWX55TRwVviCF9DrnNLxNg8gXgCgF+gXhkPPiwOtyf0Lf5XagUiEPnTkhg0EjI3Anzg5woLuGCz+QLwM0F76/fjX8I6XOgBjpXGXDMDEccuM2AY7P+4PWlMqWmWX9Qn93yvj7w5SX+PnCGH3/9UdmQ/IFfXpuCXbJYw80Bxwy7ZPljFEbGvUtdKurnQ0CVpTTEahgydzilpr88u2T5++Cl1KxeX4b6g2/zy4yHjCPnRlW/QDwSYFnlPs0uriI3W3ng9vyNgw8B/j4HWUrM7/3v2Z9Sk2GXrKC5Y16g5v+MsLnjP5QAjktWvlogzuQKwXtzhAVqo2BzE+GdAH4UwC84P5oH8LG1bNRmY6CvHccntQZ6f1+iQnvmZ39fAqtlhZNO4MvIuE7BZWbHqCifqtQqhWmHAS/4wC9wZ3I6kMMvZACO1so3ibIhGyj/rMpFxMn1GbbB+YNe+BTtPxUD4YGHGcdE7N8QowTiZEsD0h2Vk3qgNxGYXzsTcmhggdhv6uLNImwx9gt7YSZf/syga9KDTL6AkToqQCD2m3x1/cGWhkyu2uQLADu26NRR/sU1KHgF8FJH+Q9KYxNz7i2K1X2uLr+0WsKJqfnQQ0aQfz+70VTXH3wIyOT1pS6myVfXH7xh8d/7D58skAbNNW5vVR8Cst6EmXy5vD91lL5RtNrkC+g+B6WOygS4oOg2BmuIM7kCetorTb66PeHjiD/fJOy20LBxpH9WvR658Q8h5U8EXF7iDwh0+5BOhB64zYBjr/5gC9TIuA449gvcYS5ZYVYA7/KS4PJ+QYzr8Nc/t7SKMxcWqqwAus/BLlmZfLXbja6fAw+r3/Pe3mqBOyx9XzZXRH2AwM0uWWHjyD+fwzJM+VNq+vvgv81vpVTG2GQxuM+OS9aEzyUrSMMd1edMzrvUpbI9wa59vI/457N3eUnw+hLUpiAtuj+lpr+8XyBWSmE0wPKh26gtUKemqw/ctezNGwkbF47nK6V+A86FKk4WjqboPxFqYV9vO05Pz+PJMxcD8z+b8EbMbhmcgSOMwbS+7pU3CJ5QYX8TlIkjTKsHBG/S/gsIKtuTxPhspXYibAPVP6vOQxx2ijbrMDesoCwCDF9vHaTxGQ6Y1G4u1QBBKcjky22qcouJEJT48hJTII46lAQtrmzyDdPGAtX+bkEmXwCht/llQ1yB6pzAoCDhsLG+2uTLfai2ZET32Z86ik2+gVq0gMCg5dWyk84x4B2EmMaDAnYAL3VU1XvOF91bFKv7UO1uxHMvaJMOupEsKGDHLO+PlGeTb9Q4MvtcLiuMhlk+QjTEYePIE4irx1FPgMkXCD5wh7ka8c/8ueDj5o7/8pLJ4hKmjFsUK9oTEmAWFHAMhFugsk7WAb/AHeaSlclFr6lhh4wgzWFQNiQ3x3SgBrr6wD2/XHmLokmY6002H6zhDnPJyuT0teWc4YcJu97avEUxqM9+l4m4ceR3yeIMP8HrRXWfoy2rUXOnuv7uEJesbK5YkVKzos8BrjRhLiK6TdUuWe44CtlH/C5ZnOEnct/JVe7NWWev9eMF7W5cP2gbAXrFyQetAICIegCE3zUq1MxAn/Z7mywuhwYQMmYqu+VVnRHgQITQPZjSJkTOjRxm8nHLpyuTvGu/1fnAhZjr8W/S7Nvr3xx0+WotlD/Xp789XKdZP1DtOgAEBx4Gpbxj+Hprc5J6iePD+ly9wY2MB5t8dflqLXo2Vww0+XL9QOWhIe6QocsUrMp7vuhe+TCTL4DQ2/yCAr+YoPK8Ifo1UNxOf6R8mMnX7Jf53ryxHawtASoPDWzyDXpGnDrKDGLxX+pi4t7m5z+UjFcHTZpt8kfKs0XJb/IF9GHLf5tfmMlX1189LoICS836gUoXi6Ac0wxHyldviMGHVe5z8DgKLh90mx+bfHsT1XocFojNXPD+C0v85f19jtLSBR24wwKOgQgLVC5Yk6n7XO2SFbWmDge4ZI3kClUZftz6AywBQYFfTNDc8awA1c8oyCWLM/wEzYW6OsL+dLVLVpjSA0Bg4OHIeAG7e9oCLbh8m5/pkpUZr06pWd1nc30J1nCb5c335t6iGFA+yCVrtVTGsQD/ZO8zqrXoet8JH0cZn0tWJled4ces3++SFRZAzPX7+xw1jtwYBeM9ewJ39Xtua2rAzi3hec43AjYC9IcBfBFAHxG9B8C3AfzpmrZqk2EKCGEp7JiEE0k+mi/i2ETR8fkK/xs3E4ez4IX56jH+TTcsctpf3tS8hfnq6vLVgg+7iAQtfEFBL/y3+wMWpqDAw6hTN/fBrN9LqxMuBJipo1wNd6igVK1F54UvUOAO8PHL5ApoC3lvHClvaoiitG5u6qiA8kGCGFB9m1+UyZf77L+8JE5o4H6a5fdECNxA5bgLM/kCXqR8Jl857oDgDZEj5U0hwzP5ho8LUyCOMvnqPlenjooUGgIsDZmcvtTFb/IFjNv8zPIhJl/Ai5Q3++y/2r26D5WaQM/kG/6exya8GA7WQEUJDf7LS3gcBc2dsHHU2dqIvmS1htu7zS9IOAw+cO/paatav/RnR1n1vPrDAo7N8n6XrDBXI/Nzs76xbd6i6K/fbDcQnOGHYZcs2z4HuWR5AcdhAnFlkb96DQAAIABJREFU+cWVEk5Oz0f0OYEpn0tWNh/sdmP2OetfX3wZfsz6zX7y1/4MP4znkmWnxACqXbI4pWb0OPIE4lJZYXQiWGPN5RdWgpVbge0J2ZvNWxSDymd940h/dvVnuBaooDW4BmXVRsImiPDTAN4FfZX3NIDXKqU+u9YN20yYJ+A4ARrQE3g0X8TRiAwcDKdq4sXp6PgshvqrffUYXrBY2AsLCHLL+ybdhbnlqgsITFg74V+YwhaZoKCXTE7fihg0qYHqDSsoVVlFH1KVGmLWOoT1eShdmTqKfSyDTHUAAvOKsgk3CPfyEt9CMxiyIdbXEfb3VWo/R3LVl7pUtMmXRsnT6gf3YThdGSmfjTD5mj/nccGXuoQ90yDTeFjgF2Dc5ufrc5DJF3ByNfdX9znM5Kv7kKgIDIqz3gz1V6aO0sG+wSZfoNqVZjXC5AvAFQ4qND7jwWZxwLjNL1c5F8JMvty3kVyA1i3svaWTOJYvurf52Yyj5ZIXGHRuZhHFEJMvtwfw+qyUCg3kAoJzwfM4ChK43dv8fOPIvEWxqg/9yapxyn0L60PF+hKhpTN/zoL8xXm9poaNoyANcdSaurO72iVrJCDDD8MuWf4+B2X4Mdtkat3DAkWZ4f5Klyz3FsWI8twOwMjwE7oeVWtLo55RX6IZW9oaq/rsT6lZ8Rk+y0GUZZX7YLpkhWW78uqvtPaenJoLTC9rlud2AOYtimH7YPChIaz+RLO+Mdm/XvhTalb0OR2870T1we+StZGw0UADQD2AFQDLNfyNYEmyRWtHUsnmQD9AP/v6Ejg2UcTR8zprQpDGjXEvC3FuGRqJ2HyA6kwcmVy4GR3w0tmY5YHwCeHXTkQFfjF+/9go9wou79+wgrIIMMP9WiDm1FE2GiWzXNSpG/AEYi4/VVzCZHE58tDgv7wkSnvLn+3XooVpoLhvpi960KUugX3O271nv+Dj+liGPCMvdZQut7BcwukL86HakqDUUVGuA0B1rma9IQZbPrgPBSNSnm9R3NcXrkXjevX/4SZfXX9l+RNT86EmX8C8vMQncEeMC79/rM04MlNHZSNMvlz/cqmMk44fpKt1i9EEcrm4ccS3hXL58dlwky9QnQvec8cKHxf+uA//LYpVffYfuHPBAcde/ZUWqPi5EzKOQsrv8rlkzSwEZ/hhgrIhZSMEJe6zfxwFZfjx+pDEZNFzycrm9F4VJnD7FTFRbjRcv65Xl4/TcLNLFtcfllKT4Uw/fmEyzELHbRo1LFCcUjMo+F73udIlKyylpld/8LgId8eszG7CKTXDyvtdssohKTX9bfJbrOLWI9MlK5MrRu7NQS5ZGwmbLBzvBPDPALYB2AHgn4jo/1vrhm02njfQg9sG7dJr70vpazS/PpLHvlR1EJcfnVmj4GpKw071QPWtR5lccKoiE1OQcW/8igxs9OpnF5EwzYGuP+EGvXin6KiFrHrDCtPecHnAO6ln85zHMnhSs1sJC4fuDYGWWvS4hY/bxCd1NvlGHXyG+nWkPN/mF6U5ADz/8WzOE4ijBG6/Fj3K5AtUR8rH9dlNHWUI3EqFP1NuE0fKs8k3ss/pytv8woJX3Pp9B6Uok69Z3nSXCjP5AjoNZMqIlHfdbiLngjfXgm5R9GOmjmKTb1z9ZuqoTEjgl1d/pRnXNfm2hWcFMm/zizt86uut263HEf+OhQb3FsWYPp90Li/RAne00FB94A4OOHbr768eR/5LXUz8LllxGm5XIM77xlGsUkKX4ww/ceuF6ZKlg2kj1tSqPhewLxUucPtvtsvkilrgDgg4BrRLVmerpyGOCjgGzAO355oYlFLT3yZ2yWJFT9w4Mi1QsePIv17ECtyVhwwvx3Twe+hoacS2zpaK9QiI7rN5aRZn+Ilbg9klizP8xI2jlZJy15ds7N5cbZncSNhok38ewC1KqXcppd4J4FYAv7i2zdp8/NUbno2//G83WZXd3+ddGBDlvuGWTyVxfHIOh87NAIheWAFP4OYbwqImhC6v09mUy8q98cu8YtfPUDrhBozFaRr077xNOuxiioryxuIdlfKO4VyqvOmOjGtNQ9iG2OC7zS/Kx5IZNlLTeVfmRi/GnDoqE7NQAp7GL5sruibfSG2szz82TuB2b/Mb50NDuMkXqL7NzzX5hmyIgOeLzuMOCNe6AVrje3Fep44Ku9Slos/GQcm9RbEmATpak+ne5meUjzL58mdw/ZxvPUzDzeU5Ut5zTYrWunPqKDb5xmnRuO0l3hBj8tITea4ecePIf5tfnMkX0OuR+0xj3Gj07xJuLnh3HEXM/6G0Tn05mi8iz5e6xNSv216IDTgGqi1QYTmmGb9LViZXfYuiH9M0HnT7XVCf2SUr7hCj6/LWi9mQS10q66/WlkbVz5ZPs88Dfe2hyiEtECcqxlHQ7br+NrFLVpwVgMuzQBwXC2TWNZIrGLcoxj9Tc2xHjdPO1kb0d7RUjKOglJr+z3CfKQcQx8x/dsnyAo6j9x12yTpzgS91id93RsaL3i2KEX0e6GuvcsnaSNgI0CcBmG+oAcDY2jRHsMHUaNkI0IMpfer72uEcgOgTKKAn3YX5FZyeXghNVWQylE64+WM5GX/Y5sD1A3qRZBeRgV4bAboYejGFiblhRaW8YziXKl+VGneK5jaZWrQoky+XZy36yHgBHS0NoT6WZp8zuaKVVp9/l80VPC1dRHlOHZXJFUIvdTHxbvPztB+xzyiVrNCi7esLF7iBykh5Nvn6L3WpqN98RjUcxDK5gnepS0R5tkJkcgXD5Bs/F0xtbNRmBejNiQXibK4YqYHi+jlS3gumteuzjfbWvM3vdIzJF9C3+e3c0oZMvuBah6I2RKDSAhVn8uX28m1+7I61JcTky/VzLngb4dAMhLY5rJoH7riAY6DSAuVlKaltHPlvUfRjumRFZfjx9zmTtxsX5jiKCrJkTJesqLsHGL9LVtyhBPDGET/Tvb3hAje3l12yeN8JSqlp1s9tsVtfPKuem1Izog+m+yOn1IyqX7cpYew74UGTZptGHZesTF5f6hKUUtOr33PJikqp6dVfvQZHlfdcsgqhtyia+F2yNho2AvQ8gENE9Aki+hsATwG4SETvJ6L3r23zhCB6E02uT2KUUMXwZnDvwXH0JoLzrZrwJL77qfP6ewuBG+DFNX5zcIOn8kWMjIdnWmDMtGthF1OYmBuWF9QUtzDpxdhNqxPTZ77Nr7i0aqml9xZXFtCjBW6vvI1Wf3uXd3lJVCohhlNHZXKFyEtdTIad8jML8SZfQI8bTvGlA7+i34EZGDSSC77UxV8/ALfPUSZfQEfK97Q3VY4LC+FtJFc0BG6bg1UxNv0jM5xOugKxzTgyA4MyOW3yjdRAGamjbITD9uYG7OxurRxHsXNBaz/dWxRj5tpwWlvEFldKdodV47bQOHcswDeOxgvoTVRf6mLCl5eMjNsd0M0Dd1zAsdmmTK7oBRxb9JldsthFJLp+b72IyvDj1p+ufEZhGX4Y1wJlOXfYJWskVwi9mTaoTSO5AopOhp/Y9SKdxMzCCvKFpdrmjtMHG9dEwF7Rwy5Z5jiyVcSE3aLoZzitrb2LKyWMTRat5ia7ZGVCbiD21w/o9SIqpSbjWaDMfSf8GbFLFo87IH5vNl2yNho2AvTdAP4QwPcAPAzgjwA8COCQ80+4yrA2EAAORKSwY9gkPD0XHrhmwmX+48lzACxcPpwJ851jk87mED0h+GrVbK6AbD46qBHwfPx4U9+xJfoUDXgbVlTKu4ryjp/1f526CACRZmvAOzR8OzuJwmJ4FgGzPYAnHMYtfOblJew6ELUhmpeXRF3qYjLkaJRtBCv+/dTcMr4/NqX/PqY8//4HJy/EmnzNz2ctV9wzNVNHZXPFSJOv+Rk8LuJMvroPSYxWaKDiN6yFlRK+lZ2MNfnq9ujfHzo3ixNWGij2FS067hLR5c3UURkLky8AN31fXGpDt3w6geOTczh8fraijeF90Lf5fSs7GWvy5foBZ1xYaLj39HiXl2QsypuXl2RzxcBbFKv7kISt1k33QVugfnDqgvt9ZHmnzd8fm8b03HL8MzVcsqIy/DCuBWpcr8FhGX4Y8za/TC46w4/bh3SiYhzZvOeZhRV875heX2zXi8dPX9QpNS0FaLbSxdXvXV6i14s4RQ9/RjZfcFNqhmX4McubGbXixuqQ45L1rewkVkrhKTXN8gBw5HzBSakZXd68vMRmfWlt4iQF+j2HpdQ0GXQCWFkojtubTZesjYZNGrtPRv27Go0Uqnnmtg6kO5pDI79N2p10M0D8wg3oAI1kSwMOnZvVmRlCAsUYzh97j6OxjhOIiQj70wk8cfoiTk2HZ1ow4YCOuMAvxtywolLeMbyYch9steh8yIgrzwLxd0antI9ljFBiXl6itW52fWbtSpzADXDqqCU8PDYda/Ll8oBhmbB8RndbjgtOHfWDUxdDL3XxM2Rs6jZjm2/nyowXMNAbH4A7lE5ibrmEh0YmIrPRmOUB+8Mnb1BfPTQeeqmLCaeOOnRuxjH52vfZeu44gUGHzs1aH1ZXywpfPTQOIF7g5vfqzp2Y9zzQm0BDHeHrI3nML5dix0VTg7ZAjYwXMGo7d5yA1IyFSwlgHLhPX4wMOHbr968vllp023HE6UEfHpuKzPDDeFkmirHBbm6bjDU47FIXk6G0dsn67rGpyAw/Zv2AfZ+Ha1yzWSB+/PRFnL4QnlKz4jP6vUOA7b6TzRVxdLyAPT1tgSk1/X1YXCnjwaN5O4HbP3csDxlfOzyuU2rGlGeB+Mj52ciUmhVtMseRRfnh/iROTM3j0NlZq715qN9zydpo2GTheDkRPUJEeSKaJqILRDR9NRonhPO/XjaMf/3vL4gVkhieSDY+0+yPBugTcZTfKsNBKfqzLCZdKoknzszEZlpg2MdvdCJec6Dbrfv7jcyEXXucMg8czaEvGe1jCXhuJQ8cyTt/bycQ33/Ezg8d0O/sB6cu4ML8SqxmgtswUVjCk2dmYn3jdP1On4/kYk2+un5d/r7DuViTLwD0OZHy97HvveWm/oDzjOIEMa7zyPmCo4GKLz/oCMTfG5uyOrhxnfcdzoXeolhZv1feRuBOOpHyts+I2/TQyISVwK3blMSxiWLkpS7++lfLCt/ITNjVn/LGxfau1lgNFN/mx32Oe89NDXXY09tuPCO79/y9sSnnUheL95xK4PT0Ao6ej9e6cf2AM3ciAo7d+o1x0dXWiL4YDTe7ZNn2WVugjPXF8j0/ceaic6mL3TiamlvGD05dsFyPvHExmAq/e8Csn8s3N8Qrbnocl6xaxsVQOoEHj+YjU95VtCmllRInLRU9HA/03WOTls/UGxdRKTW99njlo1JqMm1N2iWrlvVlMJXEN7MTkSk1TdgCpS91sRtHpbLC10fyNe3N2Q3oxmHjwnEngN8AsB1AH4Be539hHeGcjbbwxLMR3ABvcbFZNMzyNtoYf712p1xdvlQOv5jChDU4y6vlWF86wMulGpe2h+Hb/BZWSrE+lsxgWqcfBOwWsuF00i1vq4EGoIMmbTQBZnmLZ8S3+c0vl2JNvoAXKT+/XLIy+eo+1PaMBh2XCf46vn6zz/aHDNvybI2ZXy5ZmXz5M+aXS1YaKED3gZ+R7aa+UlJWJl/AE4jnLYVPjpS3Lc+Xl8wvl6xMvtwH7nOcyRfQB/RaxxEALKyE36JY2Z7axgVboOaXSxhKxR9W2SVrfrkUm+HHbVOqtrlzKeMIsF8vBo3yNvX3Jpqwpa0R88ulyAw//s+YXy7FZvhhKvts94wWV8qxGX689phzp8b1xWJcszV5frkUm+GH4bkQl+HHLV/jGjzkCMRxGX7M+gHuc3x5dsnyX/W+EbARoM8AeFwptaKUKvG/tW6YcGX50eEUrt/agessfKYBbxO1ESYBbyGIy0bBsEAYlVzfxJzINtoP3rDMtkXBfta6vOWhIeVp6a3KO+1gU6Jteds2meVtNjhOHeX/2zD4Nj/b8mY5G5Mv4I03G9chs7z/6/D2GAc3i/KcOsq2vFnOdu7wM7Ux+Zr122igzPL+r8PgSHnArg8tjfWuL3mtfbY/0OtyNiZfXb/xni3mp3lAtTms8oEbsOsDW6B0eVulhDeOallTOyIudQmq3/zbyPrNcWRRnl2y/H8bhmn5tB5HTrn9MRl+GK4/6hbFivr7a1svzPXFpg8J073Sci54a7DlOHLK747J8OOvP+pSl4r6axxH7JJlti0KdsnaiJk4bATo3wHw70T020T0W/xvrRsmXFmev78X97ztNqsJBADXb9OC9jO2dVqV50lUq5ARlVzfhH38oi6mMDE3LNvFuNY+mIcGG4ZrLM9Cs43JF/CCM/Vn1LZh1d7nGgVoy0MG12+rgeJnaWPyBbxIefNv49uUqKl8rdYb1zpU4zONutTFhAViW4GbI+VrahMfJm3HUeoSx1GN4zTqUheTCoHYYqyaB24bLZpuU63vucZxZMxNG4Gb2xGX4Ydhlyzzb6Nglyzb8ma5Wt9zrXMz6hZFE7Z2xKXUZNgl61LaVPt6VOs4qm1uxqXUZAb62t212mZvZpcs3Tb797wRL1OxEaDfA6AEoAvadYP/CU9jnru3G1/8v5+PF+zvsSo/lE5ge1crXrC/16p8KqkzKFxneermK8BtzVaAt9DYTGqgdgGatRO2moNaFz6OlLfdEDlSvrO1MTLHtImr8bHVfvAzqlFzaON2Y5a3fUYsEA+m7QRu3ZaktckXuIw+16qNtSzPkfK25VkgthW4dVsS1hoowHs2a9Vn75natWd3dxuaGuqs62eB2Fbg1m2pcb24RGFyreZmukO7ZMXl7WeICMPppFWGn0ttkzuOLNeLWi0ZgzWOo85W7ZIVl1LTZKg/aZXhxyyv27TG48iyPAvEtvWzS1ZcSk0TbouNyweXP31hHgvLG8v5waa3KaXUc9a8JcKGgojwnN1brMu3NTXgO+94cU31f/pXbrXy7WPe8YrrsLhqP4F+4Xm7MZROxmYRYF5941acmp7HM7fbubncurcHP/3s7XjpdWmr8tu7WvHzz9uFn3r2dqvyRIT//qP7rTSrzC/+8B6cm1mwDi79mR/ajjqCtcD90uvT+M/j09Zj41k7u/DqG7fiZc/otyrf3d6EX37+HvzY9XbPFAB+/bYBa6EHAN546y48e9cWa4H7Nc/ahpmFlchLF0xeONyHV97Qjx/eZ3f4PNDfgZ961ja86satVuVbm+rx5tsG8NyBbqvyAPArL9hjXRYAXnfLzpoE7lffuBWnpuashb3n7+vBK2/ox+1Ddgfuvb3teO1zduAnbtpmVb6hvg5veeE+6wM6APzSD+/BzMKKdfmffc4OdLQ2xgYcMy9/Zj+eOjuDm3Z0WZW/eU83XnXDVrzEcn1JdzTjjc/dhZ+uYX35jRfuw7YuO2EYAH7uebtwenreen35qWdvw2pZWQvcL70uhYePbcUte+zG9g3bO/Gam7bh5ZbrS2drI970gj148YGUVXkA+LXbBtBuabkFgNffsgvP3NYZm+GH+Ykbt2GysBwbcMzcPqTXF1tl1VA6iZ959nbr9aWlsR5vvn0Az9llv/+/6QV7sVoqW5d/7c07sK2rxXpvvmF7J35o1xZMzy9je5N97NdaQ0qp6AJEfwbgXqXUgzVXTvRyAB8CUA/gE0qp94WUuwU6x/TrlFJfiKrz5ptvVo8++mitTREEQRAEQRCEmiCix5RSN/t/bnNE+nUA9xNRsZY0dkRUD+DDAF4B4HoAbyCi60PK/SmAr1q0RRAEQRAEQRDWFRv9uZ2doJpbAYwqpcYAgIg+C+AnARz2lftNAF8EcMslfs7V40UvWu8WCIIgCIIgPD156KH1boE1NjcRlgC8FsDvOl9vBfAsi7q3AzhtfH/G+ZkLEW0H8NMAPhZVERG9mYgeJaJHJyYmLD5aEARBEARBENaGWA00Ed0JoBHA7QD+N4B5aIE3TmMcFGXgd7j+IBzBPCooQSn1cQAfB7QPdFyb14xr6GQkCIIgCIIgrA02LhzPV0r9EBH9FwAopaaJyCbs+AyAncb3OwCc85W5GcBnHeG5F8AriWhVKXWXRf2CIAiCIAiCcNWxEaBXiKgOjvaYiHoA2OQreQTAIBHtBXAWwOsBvNEsoJTay18T0acA/IcIz4IgCIIgCMJGJtQHmohYuP4wdJBfHxG9B8C3obNmRKKUWgXwVujsGkcA/J/27j1W0rq+4/j7465YBa231eIuyGrWy2pAyemKeEMhLaBxm0bjGm/xElwjClRjUZO2appgNLSaqEhWqqYWQhB104pLS40XFGEFhV0QXUFldStrqVa80S3f/jHPynh6zu75ec5z5syc9yuZzDy/55mZ78l3z5xPnv3N87u4qnYm2Zxk87wrlyRJkkZg1utAJ7m2qo7tHj8BOInBvOZ/q6odi1fi7/I60JIkSVoMs10H+kBTOH77rb6q2gns7KMwSZIkaZwcKECvSvIXs+2sqnN7qEeSJEla0g4UoFcAhzHz5egkSZKkZelAAXpPVb1z0SqRJEmSxsCBViL0zLMkSZI0zYEC9ImLVoUkSZI0JmYN0FV1x2IWIkmSJI2DA52BliRJkjSNAVqSJElqYICWJEmSGhigJUmSpAYGaEmSJKmBAVqSJElqYICWJEmSGhigJUmSpAYGaEmSJKmBAVqSJElqYICWJEmSGhigJUmSpAYGaEmSJKmBAVqSJElqYICWJEmSGhigJUmSpAYGaEmSJKmBAVqSJElqYICWJEmSGhigJUmSpAYGaEmSJKmBAVqSJElqYICWJEmSGhigJUmSpAYGaEmSJKmBAVqSJElqYICWJEmSGvQaoJOcnOTmJLuSnD3D/pckub67fSXJMX3WI0mSJM1XbwE6yQrgA8ApwHrgxUnWTzvsVuBZVXU08C7g/L7qkSRJkhZCn2egNwC7quqWqroLuAjYOHxAVX2lqv6r27wKWNNjPZIkSdK89RmgVwO3DW3v7sZm82rgspl2JDktyfYk2/fu3buAJUqSJElt+gzQmWGsZjwweTaDAP2XM+2vqvOraqqqplatWrWAJUqSJEltVvb42ruBI4a21wA/mn5QkqOBLcApVfWfPdYjSZIkzVufZ6CvAdYlWZvkEGATsHX4gCRHApcCL6uqb/dYiyRJkrQgejsDXVX7kpwObANWABdU1c4km7v95wF/BTwE+GASgH1VNdVXTZIkSdJ8pWrGaclL1tTUVG3fvn3UZUiSJGnCJfn6TCd3XYlQkiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhoYoCVJkqQGvQboJCcnuTnJriRnz7A/Sd7f7b8+ybF91iNJkiTNV28BOskK4APAKcB64MVJ1k877BRgXXc7DfhQX/VIkiRJC6HPM9AbgF1VdUtV3QVcBGycdsxG4OM1cBXwwCSH91iTJEmSNC8re3zt1cBtQ9u7gafM4ZjVwJ7hg5KcxuAMNcCdSW5e2FLn7KHAT0b03loc9nh5sM/Lg31eHuzz5Btljx8502CfATozjNXvcQxVdT5w/kIUNR9JtlfV1KjrUH/s8fJgn5cH+7w82OfJtxR73OcUjt3AEUPba4Af/R7HSJIkSUtGnwH6GmBdkrVJDgE2AVunHbMVeHl3NY7jgJ9V1Z7pLyRJkiQtFb1N4aiqfUlOB7YBK4ALqmpnks3d/vOAzwKnAruAXwKv7KueBTLyaSTqnT1eHuzz8mCflwf7PPmWXI9T9f+mHEuSJEmahSsRSpIkSQ0M0JIkSVIDA/QcHGxJco2nJEck+XySm5LsTHJGN/7gJP+a5Dvd/YNGXavmJ8mKJNcl+edu2x5PmCQPTHJJkm91v9NPtc+TJ8lZ3ef1jiQXJvkD+zz+klyQ5PYkO4bGZu1rkrd2mezmJH86ipoN0AcxxyXJNZ72AW+qqscDxwGv73p7NnBFVa0Drui2Nd7OAG4a2rbHk+d9wOeq6nHAMQz6bZ8nSJLVwBuBqap6IoMLFGzCPk+CjwInTxubsa/d3+lNwBO653ywy2qLygB9cHNZklxjqKr2VNW13eOfM/iDu5pBfz/WHfYx4M9GU6EWQpI1wHOBLUPD9niCJHkA8EzgIwBVdVdV/RT7PIlWAvdNshK4H4O1I+zzmKuqLwJ3TBuera8bgYuq6jdVdSuDK7ltWJRChxigD2625cY1QZIcBTwZ+Brw8P3XI+/uHza6yrQA/h54C3D30Jg9niyPAvYC/9BN1dmS5FDs80Spqh8C7wV+AOxhsHbE5djnSTVbX5dELjNAH9yclhvX+EpyGPBJ4Myq+u9R16OFk+R5wO1V9fVR16JerQSOBT5UVU8GfoH/jT9xujmwG4G1wCOAQ5O8dLRVaQSWRC4zQB+cy41PsCT3ZhCeP1FVl3bDP05yeLf/cOD2UdWneXsa8Pwk32Mw/eo5Sf4RezxpdgO7q+pr3fYlDAK1fZ4sJwG3VtXeqvof4FLgeOzzpJqtr0silxmgD24uS5JrDCUJgzmTN1XVuUO7tgKv6B6/AvjMYtemhVFVb62qNVV1FIPf3X+vqpdijydKVf0HcFuSx3ZDJwI3Yp8nzQ+A45Lcr/v8PpHBd1fs82Sara9bgU1J7pNkLbAOuHqxi3MlwjlIciqDeZT7lyT/2xGXpAWQ5OnAl4AbuGd+7NsYzIO+GDiSwQf2C6tq+pcbNGYkDq8ZAAACsklEQVSSnAC8uaqel+Qh2OOJkuRJDL4oeghwC/BKBieJ7PMESfIO4EUMrqJ0HfAa4DDs81hLciFwAvBQ4MfAXwOfZpa+Jnk78CoG/w7OrKrLFr1mA7QkSZI0d07hkCRJkhoYoCVJkqQGBmhJkiSpgQFakiRJamCAliRJkhqsHHUBkqSB7vJ6V3SbfwT8L4MlqgF+WVXH9/S+RwHHV9U/9fH6kjRpvIydJC1BSf4GuLOq3rsI73UC3TWy+34vSZoETuGQpDGQ5M7u/oQkX0hycZJvJzknyUuSXJ3khiSP7o5bleSTSa7pbk/rxp+V5Bvd7bok9wfOAZ7RjZ2VZEWS93TPuz7Ja4fe+4tJPpXkxiTnJfHviKRlxykckjR+jgEeD9zBYNW9LVW1IckZwBuAM4H3AX9XVV9OciSwrXvOm4HXV9WVSQ4Dfg2czdAZ6CSnAT+rqj9Och/gyiSXd++9AVgPfB/4HPDnwCWL8lNL0hJhgJak8XNNVe0BSPJdYH+4vQF4dvf4JGB9kv3PeUB3tvlK4NwknwAurardQ8fs9yfA0Ule0G3/IbAOuAu4uqpu6d77QuDpGKAlLTMGaEkaP78Zenz30Pbd3PO5fi/gqVX1q2nPPSfJvwCnAlclOWmG1w/whqra9juDg7nS07844xdpJC07zl2TpMl0OXD6/o0kT+ruH11VN1TVu4HtwOOAnwP3H3ruNuB1Se7dPecxSQ7t9m1Israb+/wi4Mv9/yiStLQYoCVpMr0RmOq+BHgjsLkbPzPJjiTfBH4FXAZcD+xL8s0kZwFbgBuBa5PsAD7MPWe2v8rgS4c7gFuBTy3aTyRJS4SXsZMkzYmXu5OkAc9AS5IkSQ08Ay1JkiQ18Ay0JEmS1MAALUmSJDUwQEuSJEkNDNCSJElSAwO0JEmS1OD/AIY2H/xeq9N+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### Initialize\n",
    "environment.reset()\n",
    "\n",
    "## Creation of the environment via Environment.create() creates\n",
    "## a wrapper class around the original Environment defined here.\n",
    "## That wrapper mainly keeps track of the number of timesteps.\n",
    "## In order to alter the attributes of your instance of the original\n",
    "## class, like to set the initial temp to a custom value, like here,\n",
    "## you need to access the `environment` member of this wrapped class.\n",
    "## That is why you see the way to set the current_temp like below.\n",
    "environment.environment.current_temp = np.array([1.0])\n",
    "states = environment.environment.current_temp\n",
    "\n",
    "internals = agent.initial_internals()\n",
    "terminal = False\n",
    "\n",
    "### Run an episode\n",
    "temp = [environment.environment.current_temp[0]]\n",
    "while not terminal:\n",
    "    actions, internals = agent.act(states=states, internals=internals, evaluation=True)\n",
    "    states, terminal, reward = environment.execute(actions=actions)\n",
    "    temp += [states[0]]\n",
    "\n",
    "### Plot the run\n",
    "plt.figure(figsize=(12, 4))\n",
    "ax=plt.subplot()\n",
    "ax.set_ylim([0.0, 1.0])\n",
    "plt.plot(range(len(temp)), temp)\n",
    "plt.hlines(y=0.4, xmin=0, xmax=99, color='r')\n",
    "plt.hlines(y=0.6, xmin=0, xmax=99, color='r')\n",
    "plt.xlabel('Timestep')\n",
    "plt.ylabel('Temperature')\n",
    "plt.title('Temperature vs. Timestep')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
